Nov 03 20:26:32.192275 ip-10-0-138-52 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 20:26:32.192286 ip-10-0-138-52 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 20:26:32.192293 ip-10-0-138-52 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 20:26:32.192587 ip-10-0-138-52 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 20:26:42.441836 ip-10-0-138-52 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 20:26:42.441853 ip-10-0-138-52 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3fbb41f75f414e228d8c2e63bea9c03c -- Nov 03 20:28:39.414974 ip-10-0-138-52 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:28:39.783858 ip-10-0-138-52 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:39.783858 ip-10-0-138-52 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:28:39.783858 ip-10-0-138-52 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:39.783858 ip-10-0-138-52 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:28:39.789122 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789042 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.789122 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789118 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.789122 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789123 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.789122 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789127 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789130 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789134 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789137 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789139 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789142 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789145 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789148 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789151 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789153 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789156 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789159 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789161 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789164 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789167 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789169 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789172 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789175 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789185 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.789288 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789188 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789191 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789193 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789197 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789200 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789203 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789205 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789209 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789214 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789217 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789220 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789223 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789225 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789228 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789231 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789234 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789236 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789239 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789242 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.789755 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789244 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789247 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789249 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789252 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789254 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789257 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789260 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789262 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789265 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789274 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789277 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789280 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789282 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789285 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789289 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789292 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789295 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789297 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789301 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789304 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.790244 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789306 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789309 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789312 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789314 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789317 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789319 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789322 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789325 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789327 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789330 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789333 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789335 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789338 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789341 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789343 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789346 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789349 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789351 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789355 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789359 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.790731 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789362 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789365 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789373 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789376 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789378 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789382 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789385 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789388 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789390 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789393 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789395 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789398 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789402 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789404 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789407 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789409 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789412 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789414 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789417 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789419 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.791195 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789422 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789425 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789428 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789430 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.789433 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790208 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790215 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790219 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790224 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790227 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790229 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790232 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790234 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790237 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790240 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790246 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790249 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790252 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790255 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.791678 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790258 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790260 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790265 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790269 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790272 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790276 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790278 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790281 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790283 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790286 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790288 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790291 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790294 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790297 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790300 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790302 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790305 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790308 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790310 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.792141 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790313 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790315 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790318 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790321 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790323 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790326 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790329 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790332 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790335 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790337 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790340 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790343 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790346 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790348 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790351 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790353 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790356 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790358 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790361 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790363 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.792592 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790366 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790368 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790371 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790374 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790376 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790379 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790381 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790384 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790387 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790390 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790393 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790395 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790398 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790401 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790403 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790406 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790408 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790411 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790414 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.793115 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790416 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790419 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790421 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790423 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790426 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790429 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790432 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790435 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790437 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790440 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790442 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790445 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790447 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790450 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790452 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790455 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790457 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790460 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790463 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.793565 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790465 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790468 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790470 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790473 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790475 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790477 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790480 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790482 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790486 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790488 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790491 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790494 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790496 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790499 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.790502 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791005 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791012 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791015 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791018 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791021 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.794047 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791024 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791026 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791029 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791032 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791034 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791037 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791040 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791043 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791046 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791049 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791051 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791054 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791056 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791059 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791062 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791064 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791067 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791069 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791072 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.794516 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791075 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791078 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791080 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791083 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791085 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791088 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791090 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791094 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791096 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791099 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791102 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791105 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791108 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791111 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791113 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791116 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791118 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791121 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791124 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791128 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.794976 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791130 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791133 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791136 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791138 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791144 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791147 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791150 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791152 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791156 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791160 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791162 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791165 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791168 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791171 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791173 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791176 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791178 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791181 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791183 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.795429 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791186 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791188 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791191 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791193 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791196 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791199 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791202 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791204 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791207 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791210 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791212 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791215 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791218 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791220 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791223 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791226 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791228 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791231 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791234 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.795914 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791236 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791239 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791241 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791244 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791246 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791249 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791251 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791254 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791257 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791259 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791262 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791264 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791267 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791270 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791272 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791275 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791277 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791280 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791282 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791285 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.796355 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791287 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791292 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791296 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.791299 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792104 2572 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792114 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792120 2572 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792125 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792130 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792133 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792138 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792142 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792145 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792149 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792152 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792156 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792159 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792162 2572 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792165 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792168 2572 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792171 2572 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792174 2572 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792179 2572 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:28:39.796916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792182 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792185 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792188 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792192 2572 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792196 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792199 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792202 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792205 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792208 2572 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792213 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792216 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792219 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792222 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792229 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792232 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792235 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792238 2572 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792241 2572 flags.go:64] FLAG: --enable-server="true" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792244 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792248 2572 flags.go:64] FLAG: --event-burst="100" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792252 2572 flags.go:64] FLAG: --event-qps="50" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792254 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792257 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792260 2572 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:28:39.797455 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792264 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792267 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792270 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792273 2572 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792276 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792279 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792282 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792285 2572 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792288 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792291 2572 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792294 2572 flags.go:64] FLAG: --feature-gates="" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792302 2572 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792305 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792309 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792312 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792315 2572 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792318 2572 flags.go:64] FLAG: --help="false" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792321 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792325 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792328 2572 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792331 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792336 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792340 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:28:39.798116 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792343 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792345 2572 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792348 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792351 2572 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792354 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792358 2572 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792360 2572 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792364 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792367 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792370 2572 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792373 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792375 2572 flags.go:64] FLAG: --lock-file="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792378 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792381 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792384 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792390 2572 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792392 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792395 2572 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792399 2572 flags.go:64] FLAG: --logging-format="text" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792401 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792405 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792407 2572 flags.go:64] FLAG: --manifest-url="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792411 2572 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792415 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792418 2572 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:28:39.798746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792422 2572 flags.go:64] FLAG: --max-pods="110" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792425 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792428 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792432 2572 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792435 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792439 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792442 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792445 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792452 2572 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792455 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792458 2572 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792461 2572 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792464 2572 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792467 2572 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792470 2572 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792473 2572 flags.go:64] FLAG: --port="10250" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792476 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792479 2572 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f38ed42849de5b04" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792482 2572 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792485 2572 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792488 2572 flags.go:64] FLAG: --register-node="true" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792491 2572 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792495 2572 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792498 2572 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792501 2572 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792504 2572 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:28:39.799325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792508 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792511 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792514 2572 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792516 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792519 2572 flags.go:64] FLAG: --runonce="false" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792522 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792525 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792528 2572 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792531 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792534 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792538 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792543 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792546 2572 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792549 2572 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792552 2572 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792555 2572 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792558 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792561 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792564 2572 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792567 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792572 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792575 2572 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792578 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792582 2572 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792585 2572 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:28:39.799951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792587 2572 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792590 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792594 2572 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792597 2572 flags.go:64] FLAG: --v="2" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792601 2572 flags.go:64] FLAG: --version="false" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792606 2572 flags.go:64] FLAG: --vmodule="" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792610 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.792613 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792775 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792780 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792783 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792786 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792789 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792791 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792794 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792797 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792799 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792802 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792808 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792811 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792813 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.800556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792816 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792818 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792821 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792824 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792827 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792829 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792832 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792835 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792838 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792840 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792843 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792845 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792848 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792851 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792853 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792856 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792858 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792861 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792863 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792866 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.801078 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792869 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792872 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792874 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792877 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792880 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792883 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792885 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792888 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792892 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792898 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792902 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792905 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792908 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792911 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792914 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792916 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792919 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792922 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792924 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.801556 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792927 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792930 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792932 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792935 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792937 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792940 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792943 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792945 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792948 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792951 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792953 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792956 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792958 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792961 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792965 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792969 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792972 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792975 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792978 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792981 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.802072 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792983 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792986 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792991 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792994 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.792998 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793000 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793003 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793006 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793008 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793011 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793013 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793016 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793018 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793021 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793023 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793026 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793028 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793031 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793034 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.802543 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793036 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793039 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793041 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793044 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793046 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793049 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793051 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793054 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793057 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793059 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793062 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793065 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793067 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793070 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793072 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.793100 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:39.803067 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793135 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793139 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793143 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793146 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793149 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793152 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793154 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793157 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793160 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793163 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793166 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793168 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793171 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793174 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793177 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793180 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793185 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793187 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793190 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.803439 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793193 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793196 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793199 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793202 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793205 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793208 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793211 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793213 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793216 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793218 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793221 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793223 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793227 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793231 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793233 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793236 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793239 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793242 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793245 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793247 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.803911 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793250 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793254 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793258 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793260 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793263 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793265 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793268 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793271 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793274 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793276 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793279 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793281 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793284 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793286 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793289 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793291 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793294 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793297 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793299 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.804383 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793302 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793304 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793306 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793309 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793311 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793314 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793319 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793321 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793324 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793327 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793330 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793332 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793335 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793338 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793340 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793343 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793346 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793348 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793351 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793353 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.804850 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793356 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793359 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793361 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793363 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793366 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793369 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793371 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793374 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793376 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793379 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793381 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793384 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793386 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793389 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793391 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793394 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793396 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793400 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793404 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793407 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.805329 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793409 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793412 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793414 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793418 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793420 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793423 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793426 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.793428 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.799863 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.799878 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.800436 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.800442 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800506 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800512 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800515 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800519 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800521 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800526 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800530 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.805848 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800533 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800536 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800538 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800541 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800544 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800546 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800549 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800552 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800554 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800557 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800560 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800563 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800566 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800569 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800571 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800574 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800578 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800580 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800583 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800585 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.806299 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800588 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800590 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800594 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800597 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800600 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800603 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800605 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800608 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800611 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800614 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800616 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800631 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800634 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800637 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800640 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800643 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800645 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800648 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800651 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800653 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.806843 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800656 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800658 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800661 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800663 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800666 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800669 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800671 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800674 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800677 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800681 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800685 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800688 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800691 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800694 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800697 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800699 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800702 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800705 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800707 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.807345 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800710 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800712 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800715 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800717 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800720 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800722 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800725 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800727 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800730 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800733 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800735 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800738 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800740 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800743 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800745 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800748 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800750 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800753 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800756 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.807812 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800759 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800761 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800764 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800766 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800769 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800772 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800775 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800777 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800780 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800782 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800785 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800788 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800790 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800793 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800795 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800798 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800800 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800803 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800805 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800807 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.808293 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800810 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.800836 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800973 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800977 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800980 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800983 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800986 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800988 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800991 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800994 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800996 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.800999 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801002 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801004 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801007 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801009 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.808770 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801012 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801015 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801018 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801020 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801023 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801025 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801028 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801031 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801033 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801036 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801038 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801041 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801043 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801046 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801048 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801051 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801053 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801056 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801058 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801061 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.809200 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801063 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801066 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801068 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801071 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801073 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801076 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801078 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801081 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801085 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801090 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801093 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801096 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801098 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801101 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801104 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801107 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801110 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801112 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801115 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.809753 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801117 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801120 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801122 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801125 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801127 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801130 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801133 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801135 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801138 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801140 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801143 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801145 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801148 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801151 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801153 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801156 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801160 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801163 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801166 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.810199 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801169 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801172 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801175 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801177 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801180 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801183 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801185 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801188 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801190 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801193 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801196 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801198 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801201 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801203 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801206 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801208 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801211 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801214 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801216 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801219 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.810676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801222 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801224 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801227 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801229 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801232 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801234 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801237 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801239 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801242 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801244 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801247 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801249 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801251 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801254 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.801276 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801310 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:28:39.811225 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801315 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801318 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801321 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801323 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801326 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801329 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801333 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801335 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801338 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801341 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801343 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801346 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801348 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801350 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801357 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801359 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801362 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801365 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801368 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801371 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:28:39.811676 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801373 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801376 2572 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801378 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801381 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801383 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801386 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801388 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801391 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801394 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801396 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801399 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801401 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801405 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801408 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801411 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801413 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801416 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801418 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801421 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801424 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:28:39.812277 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801427 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801431 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801434 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801438 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801440 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801443 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801447 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801449 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801452 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801455 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801458 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801461 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801464 2572 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801466 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801469 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801471 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801474 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801476 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801479 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:28:39.813001 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801482 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801484 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801486 2572 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801489 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801493 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801497 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801500 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801502 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801505 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801509 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801512 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801515 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801518 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801521 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801524 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801527 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801529 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801532 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801534 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:28:39.813729 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801538 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801541 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801544 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801546 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801549 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801552 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801554 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801557 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801560 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801563 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801565 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801569 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801571 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801574 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801576 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801579 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801581 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801585 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801589 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801591 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:28:39.814395 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801594 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801596 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801599 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801601 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801604 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801606 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:39.801609 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.801668 2572 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.803649 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.804463 2572 server.go:1014] "Starting client certificate rotation" Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.804555 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.804594 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.806389 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:39.814879 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.807823 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:28:39.819582 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.819563 2572 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:28:39.823082 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.823066 2572 log.go:25] "Validated CRI v1 image API" Nov 03 20:28:39.823838 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.823817 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:28:39.826370 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.826345 2572 fs.go:135] Filesystem UUIDs: map[23a14c2e-5782-43ad-b0dd-97fcc04a8e92:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 94badfe8-46b9-4452-84f2-53d432bfcfc0:/dev/nvme0n1p3] Nov 03 20:28:39.826415 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.826371 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 20:28:39.831563 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.831454 2572 manager.go:217] Machine: {Timestamp:2026-11-03 20:28:39.830042278 +0000 UTC m=+0.307573757 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099291 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21a0d4b581f7738a2eace6648a1226 SystemUUID:ec21a0d4-b581-f773-8a2e-ace6648a1226 BootID:3fbb41f7-5f41-4e22-8d8c-2e63bea9c03c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:fa:ef:29:0b Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:fa:ef:29:0b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:3f:8d:62:fa:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:28:39.831563 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.831563 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:28:39.831697 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.831675 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:28:39.835047 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.835024 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:28:39.835169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.835049 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-52.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:28:39.835210 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.835171 2572 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:28:39.835210 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.835179 2572 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:28:39.835265 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.835212 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:39.835265 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.835228 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:28:39.836258 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.836248 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:28:39.840485 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.840473 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:28:39.840603 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.840593 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:28:39.841395 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.841386 2572 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:28:39.841433 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.841399 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:28:39.841433 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.841410 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:28:39.841433 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.841419 2572 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:28:39.841433 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.841427 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:28:39.841746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.841734 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:39.844201 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.844179 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:28:39.850865 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.850851 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:28:39.851742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851730 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:28:39.851785 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851760 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:28:39.851908 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851898 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:28:39.851936 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851914 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:28:39.851936 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851920 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:28:39.851936 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851926 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:28:39.851936 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851931 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:28:39.851936 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.851937 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:28:39.852764 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.852755 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:28:39.852804 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.852765 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:28:39.852804 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.852772 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:28:39.852804 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.852779 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:28:39.852804 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.852793 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:28:39.852804 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.852801 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:28:39.854454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.854444 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:28:39.854454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.854454 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:28:39.858175 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.858163 2572 server.go:1263] "Started kubelet" Nov 03 20:28:39.858300 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.858253 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:28:39.858333 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.858318 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:28:39.858371 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.858263 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:28:39.858869 ip-10-0-138-52 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:28:39.859916 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.859893 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:28:39.864810 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.864789 2572 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:28:39.865444 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:39.865426 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 20:28:39.868199 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.868183 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:39.868777 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.868765 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:28:39.868898 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:39.868846 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" Nov 03 20:28:39.869534 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869516 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:28:39.869534 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869537 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:28:39.869686 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869515 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:28:39.869686 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869656 2572 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:28:39.869686 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869663 2572 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:28:39.869814 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869758 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:28:39.869814 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869773 2572 factory.go:55] Registering systemd factory Nov 03 20:28:39.869814 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869781 2572 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:28:39.869989 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869978 2572 factory.go:153] Registering CRI-O factory Nov 03 20:28:39.870041 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.869991 2572 factory.go:223] Registration of the crio container factory successfully Nov 03 20:28:39.870041 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.870007 2572 factory.go:103] Registering Raw factory Nov 03 20:28:39.870041 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.870022 2572 manager.go:1199] Started watching for new ooms in manager Nov 03 20:28:39.871082 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.871068 2572 manager.go:319] Starting recovery of all containers Nov 03 20:28:39.881932 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.881917 2572 manager.go:324] Recovery completed Nov 03 20:28:39.887407 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.887390 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:39.889268 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889246 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:39.889339 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889286 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:39.889339 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889302 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:39.889960 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889945 2572 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:28:39.889960 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889957 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:28:39.890070 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889974 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:28:39.890070 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.889991 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:39.890662 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.890643 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:39.890737 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.890671 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:39.890737 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.890681 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:39.892269 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.892256 2572 policy_none.go:50] "Start" Nov 03 20:28:39.892316 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.892272 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:28:39.892316 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.892284 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:28:39.894253 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.894241 2572 policy_none.go:44] "Start" Nov 03 20:28:39.923208 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.923172 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:28:39.927052 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927038 2572 manager.go:341] "Starting Device Plugin manager" Nov 03 20:28:39.927125 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:39.927071 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:28:39.927125 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927082 2572 server.go:87] "Starting device plugin registration server" Nov 03 20:28:39.927321 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927309 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:28:39.927376 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927324 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:28:39.927426 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927393 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:28:39.927501 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927485 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:28:39.927501 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:39.927499 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:28:39.928071 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:39.928052 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:28:39.928152 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:39.928115 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" Nov 03 20:28:40.006496 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.006476 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-52.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 20:28:40.006496 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.006482 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-52.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 20:28:40.028213 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.028193 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:40.029108 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.029093 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:40.029192 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.029118 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:40.029192 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.029129 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:40.029192 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.029159 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.061515 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.051260 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-52.us-west-2.compute.internal.18e4a2f3dc5d0147 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-52.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-52.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:39.858135367 +0000 UTC m=+0.335666845,LastTimestamp:2026-11-03 20:28:39.858135367 +0000 UTC m=+0.335666845,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-52.us-west-2.compute.internal,}" Nov 03 20:28:40.068297 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.068282 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:28:40.068360 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.068339 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9ftcl" Nov 03 20:28:40.091563 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.091535 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:28:40.092703 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.092684 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:28:40.092703 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.092703 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:28:40.092815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.092720 2572 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:28:40.092815 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.092747 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 20:28:40.171404 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.171367 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:40.172669 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.172645 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:40.193687 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.193654 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal"] Nov 03 20:28:40.193741 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.193724 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:40.194545 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.194527 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:40.194611 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.194557 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:40.194611 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.194567 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:40.196786 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.196774 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:40.196927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.196908 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.196962 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.196942 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:40.197487 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.197472 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:40.197550 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.197484 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:40.197550 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.197497 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:40.197550 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.197508 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:40.197550 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.197511 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:40.197699 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.197564 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:40.199674 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.199656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.199763 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.199680 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:28:40.200386 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.200370 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:28:40.200465 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.200394 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:28:40.200465 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.200403 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:28:40.232224 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.232206 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.238739 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.238718 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.271899 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.271875 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d46efce11f8f6dc55307823b4dc3bdd4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"d46efce11f8f6dc55307823b4dc3bdd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.271955 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.271905 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.271955 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.271921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.272564 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.272547 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:40.272907 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.272892 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.272964 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.272917 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-52.us-west-2.compute.internal\": node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" Nov 03 20:28:40.273582 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.273569 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:40.369867 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.369820 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.373006 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.372984 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d46efce11f8f6dc55307823b4dc3bdd4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"d46efce11f8f6dc55307823b4dc3bdd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.373096 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.373010 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.373096 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.373027 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.373096 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.373069 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.373096 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.373078 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d46efce11f8f6dc55307823b4dc3bdd4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"d46efce11f8f6dc55307823b4dc3bdd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.373217 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.373081 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.427362 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.427342 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:40.428699 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.428687 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.496078 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.496062 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:28:40.535657 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.535635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.542297 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.542173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:28:40.804156 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.804089 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 20:28:40.804615 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.804223 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:40.804615 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.804232 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:40.804615 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.804223 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 20:28:40.804615 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:40.804202 2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a37db1c6ee22f4efb81f40e9e9d69904-f845c31920a170ae.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-138-52.us-west-2.compute.internal.18e4a2f3de38a91e\": read tcp 10.0.138.52:56364->18.223.49.235:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-138-52.us-west-2.compute.internal.18e4a2f3de38a91e default 4831 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-52.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-52.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-52.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 20:28:39 +0000 UTC,LastTimestamp:2026-11-03 20:28:40.029133307 +0000 UTC m=+0.506664785,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-52.us-west-2.compute.internal,}" Nov 03 20:28:40.842027 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.842009 2572 apiserver.go:52] "Watching apiserver" Nov 03 20:28:40.869077 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:40.869059 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 20:28:41.073309 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.073215 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:40 +0000 UTC" deadline="2028-05-23 16:22:54.191474788 +0000 UTC" Nov 03 20:28:41.073309 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.073260 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13603h54m13.118219772s" Nov 03 20:28:41.105678 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.105659 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:28:41.105957 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.105940 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2","openshift-multus/multus-additional-cni-plugins-8rvtd","openshift-multus/multus-qxfrx","kube-system/global-pull-secret-syncer-5khcq","openshift-network-operator/iptables-alerter-qkfwr","openshift-ovn-kubernetes/ovnkube-node-w6gnn","kube-system/konnectivity-agent-l5fnw","openshift-dns/node-resolver-cwd62","openshift-image-registry/node-ca-2j4jr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-vfs2r","kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-2fhq4","openshift-network-diagnostics/network-check-target-9x8q2"] Nov 03 20:28:41.108587 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.108566 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.110911 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.110890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.111015 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.110955 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:41.113241 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.113221 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.115385 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.115369 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.117693 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.117677 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.120112 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.119957 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.120112 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.120028 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:41.122318 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.122297 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.124501 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.124482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.126773 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.126757 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.129861 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.129834 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.132154 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.132137 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.134456 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.134440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:41.134504 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.134491 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:41.161766 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.161748 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:28:41.162251 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.162234 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:28:41.162340 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.162251 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.162340 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.162279 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.162733 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.162718 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:28:41.162818 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.162780 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c6v9\"" Nov 03 20:28:41.162884 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.162851 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:28:41.166059 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.166036 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ff7gv\"" Nov 03 20:28:41.166149 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.166092 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:28:41.166271 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.166255 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.166477 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.166462 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.167085 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.167070 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:28:41.168358 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.168342 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7b8b\"" Nov 03 20:28:41.168589 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.168575 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:28:41.168720 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.168698 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.168831 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.168724 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.170519 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.170503 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:28:41.179538 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179515 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzs87\" (UniqueName: \"kubernetes.io/projected/acf17209-903f-4a4e-9baf-908e3ab9afae-kube-api-access-wzs87\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.179618 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179539 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-cni-binary-copy\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.179618 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179556 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/536dddb8-cc7a-4b68-a591-7a19d1431afe-host-slash\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.179618 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-env-overrides\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.179618 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179587 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-netns\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.179618 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-multus\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179660 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-var-lib-kubelet\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179683 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-kubelet\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179698 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-script-lib\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-systemd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysconfig\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t97p\" (UniqueName: \"kubernetes.io/projected/a7b21223-f0b4-492b-8856-233fa4b1cd58-kube-api-access-9t97p\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179773 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.179808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179798 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdln\" (UniqueName: \"kubernetes.io/projected/2ab15921-e819-4567-af4a-9d3684c054ae-kube-api-access-bwdln\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-lib-modules\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179853 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-daemon-config\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179893 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179909 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ab15921-e819-4567-af4a-9d3684c054ae-host\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179922 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-ovn\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-system-cni-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179955 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qz8f\" (UniqueName: \"kubernetes.io/projected/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-kube-api-access-9qz8f\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.179991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34740ae-e26e-4188-a405-4da96b57553e-tmp-dir\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.180037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180012 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-sys\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-system-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-k8s-cni-cncf-io\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-var-lib-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180105 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f34740ae-e26e-4188-a405-4da96b57553e-hosts-file\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180130 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-systemd-units\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-netns\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180174 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-kubernetes\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180187 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-socket-dir-parent\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180236 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-bin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180261 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-config\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-run\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180295 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-conf-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180331 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-systemd\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-tmp\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180405 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-kubelet-config\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180438 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ab15921-e819-4567-af4a-9d3684c054ae-serviceca\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180456 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-slash\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180491 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-etc-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180558 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180573 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-tuned\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlbn\" (UniqueName: \"kubernetes.io/projected/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-kube-api-access-vtlbn\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180610 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-kubelet\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-hostroot\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180663 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-multus-certs\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180687 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-etc-kubernetes\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.180815 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-node-log\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-log-socket\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180754 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpnq\" (UniqueName: \"kubernetes.io/projected/7b8419bd-5051-4126-9569-b6344e49c66e-kube-api-access-pbpnq\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180774 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cnibin\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180789 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-modprobe-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180812 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-host\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180838 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-registration-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-device-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180902 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180920 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-netd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180942 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8419bd-5051-4126-9569-b6344e49c66e-ovn-node-metrics-cert\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzg7d\" (UniqueName: \"kubernetes.io/projected/f34740ae-e26e-4188-a405-4da96b57553e-kube-api-access-rzg7d\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.180993 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eee402df-6e0a-4cac-a025-b981a2ee6329-konnectivity-ca\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181023 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-bin\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181040 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.181261 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181074 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-dbus\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181123 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181144 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-sys-fs\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181159 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eee402df-6e0a-4cac-a025-b981a2ee6329-agent-certs\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181173 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-cnibin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181194 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-socket-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181209 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgm99\" (UniqueName: \"kubernetes.io/projected/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kube-api-access-zgm99\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181233 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/536dddb8-cc7a-4b68-a591-7a19d1431afe-iptables-alerter-script\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181254 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnfbc\" (UniqueName: \"kubernetes.io/projected/536dddb8-cc7a-4b68-a591-7a19d1431afe-kube-api-access-qnfbc\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.181738 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.181269 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-conf\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.208941 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.208919 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dfjfq\"" Nov 03 20:28:41.208941 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.208929 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:28:41.208941 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.208919 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:28:41.209295 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.209276 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kkpg\"" Nov 03 20:28:41.212238 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212220 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bb4f5\"" Nov 03 20:28:41.212555 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212539 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.212667 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212644 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.212805 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212793 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.212861 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212842 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:28:41.212930 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212867 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7m98\"" Nov 03 20:28:41.212930 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212887 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:28:41.212930 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212890 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.212930 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212913 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.213133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212967 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8tnq\"" Nov 03 20:28:41.213133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212971 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.213133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.212988 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:28:41.213133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.213066 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:28:41.213133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.213079 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:28:41.213133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.213127 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qtf8\"" Nov 03 20:28:41.213590 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.213574 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:28:41.213630 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.213598 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 20:28:41.213630 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.213625 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qc2lh" Nov 03 20:28:41.282421 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-socket-dir-parent\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.282520 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282423 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-bin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.282520 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282440 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-config\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.282520 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282453 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-run\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.282520 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282468 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-conf-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.282520 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282517 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-conf-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282532 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-socket-dir-parent\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282548 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-run\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282571 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-bin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282591 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-systemd\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282608 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-tmp\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282652 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282676 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-systemd\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-kubelet-config\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ab15921-e819-4567-af4a-9d3684c054ae-serviceca\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282718 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282724 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-kubelet-config\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-slash\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282763 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282773 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-slash\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.282822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282817 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-etc-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282842 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282848 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282868 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-tuned\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282891 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlbn\" (UniqueName: \"kubernetes.io/projected/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-kube-api-access-vtlbn\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282905 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-etc-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282917 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-kubelet\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282919 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282941 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-hostroot\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282966 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-multus-certs\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.282990 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-etc-kubernetes\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283013 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-node-log\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-log-socket\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpnq\" (UniqueName: \"kubernetes.io/projected/7b8419bd-5051-4126-9569-b6344e49c66e-kube-api-access-pbpnq\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283097 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cnibin\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283122 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-modprobe-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283134 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-config\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-kubelet\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.283609 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-host\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283244 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-hostroot\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-registration-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283264 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-host\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283288 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-multus-certs\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283334 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-etc-kubernetes\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283383 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cnibin\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283389 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-registration-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283287 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-device-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283388 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283336 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-device-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283425 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-log-socket\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283466 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-node-log\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283480 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283488 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-netd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283202 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ab15921-e819-4567-af4a-9d3684c054ae-serviceca\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.284396 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283510 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8419bd-5051-4126-9569-b6344e49c66e-ovn-node-metrics-cert\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283522 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-modprobe-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283532 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg7d\" (UniqueName: \"kubernetes.io/projected/f34740ae-e26e-4188-a405-4da96b57553e-kube-api-access-rzg7d\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283542 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-netd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283553 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eee402df-6e0a-4cac-a025-b981a2ee6329-konnectivity-ca\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283597 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-bin\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283621 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-dbus\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283670 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283726 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-sys-fs\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283762 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-bin\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283787 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eee402df-6e0a-4cac-a025-b981a2ee6329-agent-certs\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-cnibin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.285308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283830 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-socket-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgm99\" (UniqueName: \"kubernetes.io/projected/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kube-api-access-zgm99\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.283858 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283874 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/536dddb8-cc7a-4b68-a591-7a19d1431afe-iptables-alerter-script\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283881 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-dbus\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfbc\" (UniqueName: \"kubernetes.io/projected/536dddb8-cc7a-4b68-a591-7a19d1431afe-kube-api-access-qnfbc\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283788 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-sys-fs\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.283941 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:41.783904878 +0000 UTC m=+2.261436359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283964 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-cnibin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283971 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-conf\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.283997 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzs87\" (UniqueName: \"kubernetes.io/projected/acf17209-903f-4a4e-9baf-908e3ab9afae-kube-api-access-wzs87\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-cni-binary-copy\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284042 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/536dddb8-cc7a-4b68-a591-7a19d1431afe-host-slash\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-env-overrides\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284092 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-netns\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284119 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-multus\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284144 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-var-lib-kubelet\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284167 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-kubelet\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-script-lib\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284218 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-systemd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284244 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysconfig\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t97p\" (UniqueName: \"kubernetes.io/projected/a7b21223-f0b4-492b-8856-233fa4b1cd58-kube-api-access-9t97p\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284298 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-multus\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284349 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-netns\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284351 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284394 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-var-lib-kubelet\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284437 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-kubelet\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284484 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-conf\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284731 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/536dddb8-cc7a-4b68-a591-7a19d1431afe-host-slash\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284781 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-systemd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-socket-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284781 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysconfig\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.286875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284841 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdln\" (UniqueName: \"kubernetes.io/projected/2ab15921-e819-4567-af4a-9d3684c054ae-kube-api-access-bwdln\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-lib-modules\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284896 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-daemon-config\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284922 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284947 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ab15921-e819-4567-af4a-9d3684c054ae-host\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.284971 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-ovn\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285003 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-lib-modules\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285021 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-ovn\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.285029 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285051 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ab15921-e819-4567-af4a-9d3684c054ae-host\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.285082 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:41.785066237 +0000 UTC m=+2.262597720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-system-cni-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qz8f\" (UniqueName: \"kubernetes.io/projected/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-kube-api-access-9qz8f\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285168 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34740ae-e26e-4188-a405-4da96b57553e-tmp-dir\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285191 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-sys\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285208 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-system-cni-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-system-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.287528 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285255 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285293 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-k8s-cni-cncf-io\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285321 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-var-lib-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285336 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-sys\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f34740ae-e26e-4188-a405-4da96b57553e-hosts-file\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285373 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-systemd-units\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285381 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-k8s-cni-cncf-io\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-netns\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285436 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285457 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-kubernetes\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285484 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285486 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f34740ae-e26e-4188-a405-4da96b57553e-hosts-file\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285294 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-system-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285501 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34740ae-e26e-4188-a405-4da96b57553e-tmp-dir\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285516 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-daemon-config\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285552 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-systemd-units\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285566 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-netns\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285626 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285685 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-cni-binary-copy\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285686 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-env-overrides\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285748 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-var-lib-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.285840 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-kubernetes\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.286772 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eee402df-6e0a-4cac-a025-b981a2ee6329-konnectivity-ca\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.286781 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-script-lib\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.286826 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-tmp\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.287011 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-tuned\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.287119 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.287212 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eee402df-6e0a-4cac-a025-b981a2ee6329-agent-certs\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.287251 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/536dddb8-cc7a-4b68-a591-7a19d1431afe-iptables-alerter-script\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.288649 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.288157 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8419bd-5051-4126-9569-b6344e49c66e-ovn-node-metrics-cert\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.343670 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.343580 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:41.343670 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.343609 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:41.343670 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.343623 2572 projected.go:196] Error preparing data for projected volume kube-api-access-dlz94 for pod openshift-network-diagnostics/network-check-target-9x8q2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:41.344012 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.343698 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94 podName:fe95bd0d-a5b5-44db-97ae-d75a6860a71a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:41.843681037 +0000 UTC m=+2.321212523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dlz94" (UniqueName: "kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94") pod "network-check-target-9x8q2" (UID: "fe95bd0d-a5b5-44db-97ae-d75a6860a71a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:41.346661 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.346567 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfbc\" (UniqueName: \"kubernetes.io/projected/536dddb8-cc7a-4b68-a591-7a19d1431afe-kube-api-access-qnfbc\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.346661 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.346607 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg7d\" (UniqueName: \"kubernetes.io/projected/f34740ae-e26e-4188-a405-4da96b57553e-kube-api-access-rzg7d\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.346661 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.346627 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgm99\" (UniqueName: \"kubernetes.io/projected/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kube-api-access-zgm99\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.346661 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.346630 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzs87\" (UniqueName: \"kubernetes.io/projected/acf17209-903f-4a4e-9baf-908e3ab9afae-kube-api-access-wzs87\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.347045 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.347027 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t97p\" (UniqueName: \"kubernetes.io/projected/a7b21223-f0b4-492b-8856-233fa4b1cd58-kube-api-access-9t97p\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.347045 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.347035 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdln\" (UniqueName: \"kubernetes.io/projected/2ab15921-e819-4567-af4a-9d3684c054ae-kube-api-access-bwdln\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.347307 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.347289 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlbn\" (UniqueName: \"kubernetes.io/projected/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-kube-api-access-vtlbn\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.347538 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.347519 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpnq\" (UniqueName: \"kubernetes.io/projected/7b8419bd-5051-4126-9569-b6344e49c66e-kube-api-access-pbpnq\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.347912 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.347895 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qz8f\" (UniqueName: \"kubernetes.io/projected/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-kube-api-access-9qz8f\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.451568 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.451525 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qxfrx" Nov 03 20:28:41.459443 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.459425 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:28:41.466608 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.466589 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:28:41.472365 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.472341 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:41.475903 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.475881 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:28:41.508091 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.508066 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cwd62" Nov 03 20:28:41.524152 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.524127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:28:41.549874 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.549847 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:28:41.554710 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.554691 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:28:41.746260 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:41.746198 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca837b7a3905083ff9db7ad20c4a2d6d.slice/crio-2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a WatchSource:0}: Error finding container 2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a: Status 404 returned error can't find the container with id 2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a Nov 03 20:28:41.748068 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:41.748046 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46efce11f8f6dc55307823b4dc3bdd4.slice/crio-8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9 WatchSource:0}: Error finding container 8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9: Status 404 returned error can't find the container with id 8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9 Nov 03 20:28:41.753383 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.753368 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:28:41.764459 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.764442 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:41.791225 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.791195 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:41.791301 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.791241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:41.791355 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.791340 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:41.791401 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.791392 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.791378694 +0000 UTC m=+3.268910162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:41.791447 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.791416 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:41.791487 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.791467 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.791447776 +0000 UTC m=+3.268979259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:41.892001 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:41.891974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:41.892463 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.892083 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:41.892463 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.892096 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:41.892463 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.892106 2572 projected.go:196] Error preparing data for projected volume kube-api-access-dlz94 for pod openshift-network-diagnostics/network-check-target-9x8q2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:41.892463 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:41.892145 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94 podName:fe95bd0d-a5b5-44db-97ae-d75a6860a71a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:42.892133383 +0000 UTC m=+3.369664851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlz94" (UniqueName: "kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94") pod "network-check-target-9x8q2" (UID: "fe95bd0d-a5b5-44db-97ae-d75a6860a71a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:42.019096 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.019043 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:42.098374 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.098329 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"d46efce11f8f6dc55307823b4dc3bdd4","Type":"ContainerStarted","Data":"8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9"} Nov 03 20:28:42.099483 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.099459 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"ca837b7a3905083ff9db7ad20c4a2d6d","Type":"ContainerStarted","Data":"2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a"} Nov 03 20:28:42.216841 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.216803 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:41 +0000 UTC" deadline="2028-03-30 03:02:07.15826112 +0000 UTC" Nov 03 20:28:42.216841 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.216839 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12294h33m24.941424395s" Nov 03 20:28:42.282322 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.282267 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:28:42.359015 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.358994 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0d106f_4b85_48cf_8c77_6b4966cfd1f3.slice/crio-68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811 WatchSource:0}: Error finding container 68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811: Status 404 returned error can't find the container with id 68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811 Nov 03 20:28:42.379445 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.379421 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b21223_f0b4_492b_8856_233fa4b1cd58.slice/crio-40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf WatchSource:0}: Error finding container 40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf: Status 404 returned error can't find the container with id 40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf Nov 03 20:28:42.440654 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.440628 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34740ae_e26e_4188_a405_4da96b57553e.slice/crio-18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7 WatchSource:0}: Error finding container 18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7: Status 404 returned error can't find the container with id 18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7 Nov 03 20:28:42.441672 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.441648 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8419bd_5051_4126_9569_b6344e49c66e.slice/crio-5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7 WatchSource:0}: Error finding container 5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7: Status 404 returned error can't find the container with id 5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7 Nov 03 20:28:42.447155 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.447136 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311c1fbb_32b5_4d7d_934b_28b68c8d48cb.slice/crio-3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93 WatchSource:0}: Error finding container 3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93: Status 404 returned error can't find the container with id 3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93 Nov 03 20:28:42.450637 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.450619 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab15921_e819_4567_af4a_9d3684c054ae.slice/crio-ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d WatchSource:0}: Error finding container ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d: Status 404 returned error can't find the container with id ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d Nov 03 20:28:42.468440 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.468422 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf17209_903f_4a4e_9baf_908e3ab9afae.slice/crio-bdef0f609171701d4402601c567d98279b272e7a2fe24309f053f170103946c7 WatchSource:0}: Error finding container bdef0f609171701d4402601c567d98279b272e7a2fe24309f053f170103946c7: Status 404 returned error can't find the container with id bdef0f609171701d4402601c567d98279b272e7a2fe24309f053f170103946c7 Nov 03 20:28:42.501418 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.501399 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536dddb8_cc7a_4b68_a591_7a19d1431afe.slice/crio-1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0 WatchSource:0}: Error finding container 1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0: Status 404 returned error can't find the container with id 1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0 Nov 03 20:28:42.640037 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:28:42.640015 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee402df_6e0a_4cac_a025_b981a2ee6329.slice/crio-cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744 WatchSource:0}: Error finding container cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744: Status 404 returned error can't find the container with id cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744 Nov 03 20:28:42.797840 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.797806 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:42.797986 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.797867 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:42.798051 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.797989 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:42.798051 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.798042 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.798024499 +0000 UTC m=+5.275555981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:42.798155 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.798107 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:42.798155 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.798144 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.798131651 +0000 UTC m=+5.275663121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:42.898938 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:42.898865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:42.899428 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.899042 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:42.899428 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.899067 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:42.899428 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.899082 2572 projected.go:196] Error preparing data for projected volume kube-api-access-dlz94 for pod openshift-network-diagnostics/network-check-target-9x8q2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:42.899428 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:42.899138 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94 podName:fe95bd0d-a5b5-44db-97ae-d75a6860a71a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:44.899119574 +0000 UTC m=+5.376651057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlz94" (UniqueName: "kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94") pod "network-check-target-9x8q2" (UID: "fe95bd0d-a5b5-44db-97ae-d75a6860a71a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:43.093520 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.093487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:43.093727 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.093521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:43.093727 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:43.093626 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:43.093727 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.093682 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:43.093910 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:43.093795 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:43.093910 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:43.093892 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:43.103478 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.103436 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qkfwr" event={"ID":"536dddb8-cc7a-4b68-a591-7a19d1431afe","Type":"ContainerStarted","Data":"1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0"} Nov 03 20:28:43.105268 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.105245 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" event={"ID":"acf17209-903f-4a4e-9baf-908e3ab9afae","Type":"ContainerStarted","Data":"bdef0f609171701d4402601c567d98279b272e7a2fe24309f053f170103946c7"} Nov 03 20:28:43.107917 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.107889 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwd62" event={"ID":"f34740ae-e26e-4188-a405-4da96b57553e","Type":"ContainerStarted","Data":"18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7"} Nov 03 20:28:43.109411 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.109377 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qxfrx" event={"ID":"a7b21223-f0b4-492b-8856-233fa4b1cd58","Type":"ContainerStarted","Data":"40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf"} Nov 03 20:28:43.110780 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.110756 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" event={"ID":"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3","Type":"ContainerStarted","Data":"68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811"} Nov 03 20:28:43.112088 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.112063 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l5fnw" event={"ID":"eee402df-6e0a-4cac-a025-b981a2ee6329","Type":"ContainerStarted","Data":"cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744"} Nov 03 20:28:43.113629 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.113581 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2j4jr" event={"ID":"2ab15921-e819-4567-af4a-9d3684c054ae","Type":"ContainerStarted","Data":"ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d"} Nov 03 20:28:43.115122 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.115086 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerStarted","Data":"3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93"} Nov 03 20:28:43.116558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.116537 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7"} Nov 03 20:28:43.217274 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.217198 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:41 +0000 UTC" deadline="2028-06-26 01:55:57.680556031 +0000 UTC" Nov 03 20:28:43.217274 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:43.217235 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14405h27m14.463326463s" Nov 03 20:28:44.816377 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:44.815919 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:44.816377 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:44.815978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:44.816377 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.816157 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:44.816377 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.816214 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:48.81619594 +0000 UTC m=+9.293727427 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:44.816377 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.816278 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:44.816377 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.816311 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:48.816299559 +0000 UTC m=+9.293831029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:44.917288 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:44.916988 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:44.917288 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.917165 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:44.917288 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.917183 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:44.917288 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.917198 2572 projected.go:196] Error preparing data for projected volume kube-api-access-dlz94 for pod openshift-network-diagnostics/network-check-target-9x8q2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:44.917288 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:44.917258 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94 podName:fe95bd0d-a5b5-44db-97ae-d75a6860a71a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:48.917240093 +0000 UTC m=+9.394771568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlz94" (UniqueName: "kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94") pod "network-check-target-9x8q2" (UID: "fe95bd0d-a5b5-44db-97ae-d75a6860a71a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:45.095062 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:45.093943 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:45.095062 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:45.094086 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:45.095062 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:45.094158 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:45.095062 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:45.094228 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:45.095062 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:45.094287 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:45.095062 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:45.094367 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:46.153169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:46.150607 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"d46efce11f8f6dc55307823b4dc3bdd4","Type":"ContainerStarted","Data":"7b7a56108c6463f30cd6abeb1ff6114cc927aeb6b43e7d948ef5e293c0f38529"} Nov 03 20:28:46.174889 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:46.174815 2572 generic.go:356] "Generic (PLEG): container finished" podID="ca837b7a3905083ff9db7ad20c4a2d6d" containerID="76fee7a26009e19fa86de139f602f20daf7925afc1fbe34d56d5674145c131bd" exitCode=0 Nov 03 20:28:46.174889 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:46.174862 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"ca837b7a3905083ff9db7ad20c4a2d6d","Type":"ContainerDied","Data":"76fee7a26009e19fa86de139f602f20daf7925afc1fbe34d56d5674145c131bd"} Nov 03 20:28:46.388022 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:46.387964 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" podStartSLOduration=5.387943093 podStartE2EDuration="5.387943093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:46.269558141 +0000 UTC m=+6.747089631" watchObservedRunningTime="2026-11-03 20:28:46.387943093 +0000 UTC m=+6.865474585" Nov 03 20:28:47.094132 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.093615 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:47.094132 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:47.093768 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:47.094132 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.093849 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:47.094132 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:47.093945 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:47.094132 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.093996 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:47.094132 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:47.094072 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:47.184693 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.184637 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"ca837b7a3905083ff9db7ad20c4a2d6d","Type":"ContainerStarted","Data":"b9a19652f447e015c201c12ab2c344164ab8d87562eef9fe26a4bebe592cc9e3"} Nov 03 20:28:47.189086 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.189053 2572 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="9c5d74b446f78eae985143f2d46d78749c757a6cee42cf85783dadef3bcec1bc" exitCode=0 Nov 03 20:28:47.189988 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.189964 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerDied","Data":"9c5d74b446f78eae985143f2d46d78749c757a6cee42cf85783dadef3bcec1bc"} Nov 03 20:28:47.431760 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:47.430842 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" podStartSLOduration=6.430821491 podStartE2EDuration="6.430821491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:47.303893765 +0000 UTC m=+7.781425259" watchObservedRunningTime="2026-11-03 20:28:47.430821491 +0000 UTC m=+7.908352985" Nov 03 20:28:48.198399 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:48.198329 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l5fnw" event={"ID":"eee402df-6e0a-4cac-a025-b981a2ee6329","Type":"ContainerStarted","Data":"fae0d2e57374fdf8428e0395616fcf9d0915331685c808d686a461bd85aa32b0"} Nov 03 20:28:48.855079 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:48.855037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:48.855255 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:48.855136 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:48.855331 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.855268 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:48.855389 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.855339 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.855318058 +0000 UTC m=+17.332849529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:48.855451 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.855414 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:48.855502 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.855451 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.855439066 +0000 UTC m=+17.332970546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:48.955724 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:48.955689 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:48.955918 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.955895 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:48.955990 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.955919 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:48.955990 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.955934 2572 projected.go:196] Error preparing data for projected volume kube-api-access-dlz94 for pod openshift-network-diagnostics/network-check-target-9x8q2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:48.956109 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:48.955993 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94 podName:fe95bd0d-a5b5-44db-97ae-d75a6860a71a nodeName:}" failed. No retries permitted until 2026-11-03 20:28:56.955973696 +0000 UTC m=+17.433505166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlz94" (UniqueName: "kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94") pod "network-check-target-9x8q2" (UID: "fe95bd0d-a5b5-44db-97ae-d75a6860a71a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:49.093332 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:49.093296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:49.093595 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:49.093296 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:49.093595 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:49.093428 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:49.093595 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:49.093463 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:49.093595 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:49.093565 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:49.093872 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:49.093622 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:49.204112 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:49.203894 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" event={"ID":"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3","Type":"ContainerStarted","Data":"e4e09f9a002820c198cfe93debe502d379f9e16b0c85d231700fe3170bb431dd"} Nov 03 20:28:49.207519 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:49.207413 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2j4jr" event={"ID":"2ab15921-e819-4567-af4a-9d3684c054ae","Type":"ContainerStarted","Data":"ba617055ba6984e575f00f2172acb6c71cfe9daa7dab0fa3fe576ab201ed8f09"} Nov 03 20:28:49.324354 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:49.323876 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l5fnw" podStartSLOduration=4.550808778 podStartE2EDuration="9.323853507s" totalImagesPullingTime="4.773044729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:48.31596171 +0000 UTC m=+8.793493202" watchObservedRunningTime="2026-11-03 20:28:49.323853507 +0000 UTC m=+9.801385000" Nov 03 20:28:51.093857 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.093544 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:51.093857 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:51.093674 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:51.093857 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.093734 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:51.093857 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:51.093802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:51.093857 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.093857 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:51.094280 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:51.093932 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:51.227501 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.227404 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:28:51.936133 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.936025 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:28:51.227425841Z","UUID":"1f524684-d7e5-4c09-8afa-da59658ed432","Handler":null,"Name":"","Endpoint":""} Nov 03 20:28:51.937804 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.937787 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:28:51.937894 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:51.937816 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:28:52.219335 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.219296 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cwd62" event={"ID":"f34740ae-e26e-4188-a405-4da96b57553e","Type":"ContainerStarted","Data":"867f9ab2b1ac911098c1e2f2462dac7ddf75e32656caf4a0edff04c876245041"} Nov 03 20:28:52.222323 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.221998 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qxfrx" event={"ID":"a7b21223-f0b4-492b-8856-233fa4b1cd58","Type":"ContainerStarted","Data":"90f0e629a5719a3d18c0588f204ee125ed511b2fe22f08bbbf24f9cafc4fa734"} Nov 03 20:28:52.224599 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.224534 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" event={"ID":"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3","Type":"ContainerStarted","Data":"44971059b16e698292a15760c02d36759339b3d48f9b9b43b7abeb5332cc86b9"} Nov 03 20:28:52.227780 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.227752 2572 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="3de40bd4ecc5acccfaf5591ef5f9f610eeba4e659874df731f4ad3a7c04197b3" exitCode=0 Nov 03 20:28:52.227897 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.227796 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerDied","Data":"3de40bd4ecc5acccfaf5591ef5f9f610eeba4e659874df731f4ad3a7c04197b3"} Nov 03 20:28:52.327338 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.327292 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cwd62" podStartSLOduration=2.22417084 podStartE2EDuration="11.327280254s" totalImagesPullingTime="9.103109414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:52.326933238 +0000 UTC m=+12.804464727" watchObservedRunningTime="2026-11-03 20:28:52.327280254 +0000 UTC m=+12.804811741" Nov 03 20:28:52.327515 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.327372 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2j4jr" podStartSLOduration=5.156341044 podStartE2EDuration="11.327365532s" totalImagesPullingTime="6.171024488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:49.324318686 +0000 UTC m=+9.801850181" watchObservedRunningTime="2026-11-03 20:28:52.327365532 +0000 UTC m=+12.804897022" Nov 03 20:28:52.830807 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.829957 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:52.830807 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.830521 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:28:52.939226 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:52.939176 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qxfrx" podStartSLOduration=3.905879982 podStartE2EDuration="12.939161416s" totalImagesPullingTime="9.033281434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:52.552821995 +0000 UTC m=+13.030353485" watchObservedRunningTime="2026-11-03 20:28:52.939161416 +0000 UTC m=+13.416692903" Nov 03 20:28:53.093496 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:53.093420 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:53.093667 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:53.093548 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:53.093667 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:53.093564 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:53.093789 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:53.093679 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:53.093789 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:53.093753 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:53.093890 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:53.093845 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:53.232327 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:53.232283 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qkfwr" event={"ID":"536dddb8-cc7a-4b68-a591-7a19d1431afe","Type":"ContainerStarted","Data":"3204426845e3d7d10fb84747bbcfb693a892ec56b174c744c244bdf4075d8858"} Nov 03 20:28:53.339557 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:53.339504 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qkfwr" podStartSLOduration=3.5437297 podStartE2EDuration="12.339490842s" totalImagesPullingTime="8.795761142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:53.339155069 +0000 UTC m=+13.816686559" watchObservedRunningTime="2026-11-03 20:28:53.339490842 +0000 UTC m=+13.817022329" Nov 03 20:28:54.235551 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:54.235473 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" event={"ID":"acf17209-903f-4a4e-9baf-908e3ab9afae","Type":"ContainerStarted","Data":"3c7ca7f52992c8bfb513234b1fcd7f48944c4b564ec5d56cb51a52874440639e"} Nov 03 20:28:54.237983 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:54.237951 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" event={"ID":"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3","Type":"ContainerStarted","Data":"769d39bddb0a439273b6038a2af9327a08e9211bfae1d15274fd9c7280dd0bb5"} Nov 03 20:28:54.240344 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:54.240317 2572 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="f78a0810cb7e7548d9f257237559b48eac86874eb7267af1c798ebba0fe7e4ba" exitCode=0 Nov 03 20:28:54.240460 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:54.240397 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerDied","Data":"f78a0810cb7e7548d9f257237559b48eac86874eb7267af1c798ebba0fe7e4ba"} Nov 03 20:28:54.353101 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:54.353047 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" podStartSLOduration=2.150105535 podStartE2EDuration="13.353031015s" totalImagesPullingTime="11.20292548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:54.352572926 +0000 UTC m=+14.830104412" watchObservedRunningTime="2026-11-03 20:28:54.353031015 +0000 UTC m=+14.830562499" Nov 03 20:28:54.468032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:54.467975 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" podStartSLOduration=4.104771783 podStartE2EDuration="14.467959698s" totalImagesPullingTime="10.363187915s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:54.46777254 +0000 UTC m=+14.945304028" watchObservedRunningTime="2026-11-03 20:28:54.467959698 +0000 UTC m=+14.945491183" Nov 03 20:28:55.093758 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:55.093723 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:55.093758 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:55.093738 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:55.093946 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:55.093866 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:55.093946 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:55.093918 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:55.094030 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:55.093992 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:55.094074 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:55.094060 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:56.247117 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:56.247038 2572 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="c25a652b56db1372a4c20b50b15b545bf3213413138834ade5e93a08298297ac" exitCode=0 Nov 03 20:28:56.247117 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:56.247083 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerDied","Data":"c25a652b56db1372a4c20b50b15b545bf3213413138834ade5e93a08298297ac"} Nov 03 20:28:56.916586 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:56.916557 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:56.916778 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:56.916648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:56.916778 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:56.916712 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:56.916778 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:56.916749 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:56.916919 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:56.916781 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.916757558 +0000 UTC m=+33.394289023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:28:56.916919 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:56.916801 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.916791365 +0000 UTC m=+33.394322842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:28:57.017087 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:57.017051 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:57.017260 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.017203 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 20:28:57.017260 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.017217 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 20:28:57.017260 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.017229 2572 projected.go:196] Error preparing data for projected volume kube-api-access-dlz94 for pod openshift-network-diagnostics/network-check-target-9x8q2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:57.017404 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.017273 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94 podName:fe95bd0d-a5b5-44db-97ae-d75a6860a71a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.017261332 +0000 UTC m=+33.494792797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlz94" (UniqueName: "kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94") pod "network-check-target-9x8q2" (UID: "fe95bd0d-a5b5-44db-97ae-d75a6860a71a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 20:28:57.092931 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:57.092896 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:57.093114 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:57.093009 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:57.093114 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.093022 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:57.093214 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.093131 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:57.093387 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:57.093363 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:57.093464 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:57.093438 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:58.252704 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:58.252462 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"9f999b78e50227e867c1ecec1214636fbd9160f079bc88c535696a8e121e1e0f"} Nov 03 20:28:59.093093 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.092909 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:28:59.093189 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.092938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:28:59.093189 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:59.093158 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:28:59.093268 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.092974 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:28:59.093268 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:59.093239 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:28:59.093324 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:28:59.093290 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:28:59.256986 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.256926 2572 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="312c9b98175a36960fe4c92de1f2830a79bdcbda03fb87dc5ac4c6f18c48f7c3" exitCode=0 Nov 03 20:28:59.257393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.256998 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerDied","Data":"312c9b98175a36960fe4c92de1f2830a79bdcbda03fb87dc5ac4c6f18c48f7c3"} Nov 03 20:28:59.260180 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.260155 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"6fcbe38520f08c410e1fbab35215a3f8d00cb7782f6adb0d81dfb5f646257269"} Nov 03 20:28:59.260234 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.260209 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"9566799e1d68af4a69b5060343b8f9569352e614bacae51841664b56675641e1"} Nov 03 20:28:59.260280 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.260240 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"ac933cdbb08cf55b3f52af31aa5e1de8b02aeb55c418644d85de30c9352829b7"} Nov 03 20:28:59.260280 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.260269 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"8cc62cd8b5fb001d967a60806a1ee9df4f1582d3c0ba94e0722b9dbd9b644a45"} Nov 03 20:28:59.260348 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:28:59.260296 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"15036fe03576bf03fad16e3ebf5fe34fae120585c571738db25d8a96bee65052"} Nov 03 20:29:00.265115 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:00.265086 2572 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="e13b245d5c2e02cb8e94d879f3fc89b4a00087dcf4c56c987509ee1e8c398793" exitCode=0 Nov 03 20:29:00.265585 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:00.265129 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerDied","Data":"e13b245d5c2e02cb8e94d879f3fc89b4a00087dcf4c56c987509ee1e8c398793"} Nov 03 20:29:01.092988 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:01.092948 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:01.092988 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:01.092976 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:01.093192 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:01.093049 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:01.093192 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:01.092960 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:01.093192 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:01.093137 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:01.093297 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:01.093196 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:01.270639 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:01.270586 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" event={"ID":"311c1fbb-32b5-4d7d-934b-28b68c8d48cb","Type":"ContainerStarted","Data":"dbce78daac5efe52629e881d0a03b9c4ebdc3b6ec969dc03618ce170ce8b47ec"} Nov 03 20:29:01.273567 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:01.273546 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"e62d9dc30dabb4f52afd700b3aa5a3489ce966c00b4589df285d4d72bed1c6b9"} Nov 03 20:29:01.386350 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:01.386302 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8rvtd" podStartSLOduration=6.212588163 podStartE2EDuration="20.386290185s" totalImagesPullingTime="14.173702022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:01.385561291 +0000 UTC m=+21.863092778" watchObservedRunningTime="2026-11-03 20:29:01.386290185 +0000 UTC m=+21.863821669" Nov 03 20:29:03.093475 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.093443 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:03.094178 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.093523 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:03.094178 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.093548 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:03.094178 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:03.093642 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:03.094178 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:03.093685 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:03.094178 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:03.093734 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:03.279931 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.279906 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" event={"ID":"7b8419bd-5051-4126-9569-b6344e49c66e","Type":"ContainerStarted","Data":"49c9a46f7e18fcf0c2c24e3151163a648e53ce421f18406522e1888b13a6f993"} Nov 03 20:29:03.280118 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.280102 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:03.280268 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.280251 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:03.280342 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.280313 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:03.294802 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.294783 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:03.294904 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.294878 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:03.540423 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:03.540375 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" podStartSLOduration=7.794625192 podStartE2EDuration="23.54036066s" totalImagesPullingTime="15.745735468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:03.540300308 +0000 UTC m=+24.017831794" watchObservedRunningTime="2026-11-03 20:29:03.54036066 +0000 UTC m=+24.017892146" Nov 03 20:29:05.093853 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.093822 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:05.094365 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.093890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:05.094365 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.093976 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:05.094365 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:05.093999 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:05.094365 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:05.094045 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:05.094365 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:05.094154 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:05.357234 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.357165 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5khcq"] Nov 03 20:29:05.357356 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.357293 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:05.357394 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:05.357368 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:05.361278 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.361251 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9x8q2"] Nov 03 20:29:05.361422 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.361367 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:05.361489 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:05.361465 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:05.361922 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.361897 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfs2r"] Nov 03 20:29:05.362041 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:05.361978 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:05.362099 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:05.362045 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:07.093518 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:07.093442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:07.093944 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:07.093557 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:07.093944 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:07.093566 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:07.093944 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:07.093596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:07.093944 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:07.093727 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:07.093944 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:07.093813 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:09.093546 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:09.093521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:09.093956 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:09.093555 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:09.093956 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:09.093563 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:09.093956 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:09.093604 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:09.093956 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:09.093718 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:09.093956 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:09.093779 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:11.093675 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.093583 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:11.093675 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.093649 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:11.093675 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.093658 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:11.094100 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.093725 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9x8q2" podUID="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" Nov 03 20:29:11.094100 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.093772 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5khcq" podUID="c947407f-1cc6-4c01-997e-99e19a3c139c" Nov 03 20:29:11.094100 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.093853 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfs2r" podUID="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" Nov 03 20:29:11.274407 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.274375 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:11.274589 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.274514 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:29:11.274905 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.274885 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:11.391257 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.391235 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeReady" Nov 03 20:29:11.391404 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.391350 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 20:29:11.491219 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.491189 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k2tz8"] Nov 03 20:29:11.495164 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.495147 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:11.499288 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.499267 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-48l5q"] Nov 03 20:29:11.502901 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.502849 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8bznz"] Nov 03 20:29:11.502974 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.502956 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:11.509577 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.509558 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt"] Nov 03 20:29:11.509705 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.509690 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:11.512845 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.512829 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ljjts"] Nov 03 20:29:11.512944 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.512929 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:11.515852 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.515837 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-627kh"] Nov 03 20:29:11.515971 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.515955 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.519085 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.519068 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7"] Nov 03 20:29:11.519211 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.519193 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.522069 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.522052 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-75cb4f9586-ps4d8"] Nov 03 20:29:11.522170 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.522157 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.525081 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.525065 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zd9mv"] Nov 03 20:29:11.525175 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.525161 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.528078 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.528053 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76f9745bc9-fxgzq"] Nov 03 20:29:11.528208 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.528190 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.530951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.530934 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b"] Nov 03 20:29:11.531048 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.531035 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.534044 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.534028 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2b69b"] Nov 03 20:29:11.534145 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.534131 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.537556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.537538 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7"] Nov 03 20:29:11.537775 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.537650 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:11.540488 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.540473 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-595485878c-mqdnk"] Nov 03 20:29:11.540595 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.540582 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.543527 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.543510 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg"] Nov 03 20:29:11.544672 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.543888 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.545411 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.545394 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.545411 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.545404 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.546932 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.546917 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sd2tb"] Nov 03 20:29:11.547050 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.547034 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:11.549798 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.549781 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bw59n"] Nov 03 20:29:11.549896 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.549882 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.552653 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.552636 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.555830 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.555812 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k2tz8"] Nov 03 20:29:11.564380 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.564362 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-48l5q"] Nov 03 20:29:11.565141 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.565116 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pcxgh\"" Nov 03 20:29:11.570722 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.570707 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt"] Nov 03 20:29:11.573661 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.573644 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8bznz"] Nov 03 20:29:11.603399 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.603376 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7"] Nov 03 20:29:11.608637 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.608502 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-627kh"] Nov 03 20:29:11.610313 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.610293 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zd9mv"] Nov 03 20:29:11.610761 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.610744 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76f9745bc9-fxgzq"] Nov 03 20:29:11.618779 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618744 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j877w\" (UniqueName: \"kubernetes.io/projected/5e448fa5-814c-427f-854e-02708388f82a-kube-api-access-j877w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.618863 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618796 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-config\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.618863 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618824 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.618863 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618852 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnlx\" (UniqueName: \"kubernetes.io/projected/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-kube-api-access-swnlx\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618868 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618883 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618899 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d942d\" (UniqueName: \"kubernetes.io/projected/6ab99b87-827c-4aba-a47e-b86a847ca383-kube-api-access-d942d\") pod \"volume-data-source-validator-58c9f5f46d-b74gt\" (UID: \"6ab99b87-827c-4aba-a47e-b86a847ca383\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618918 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q4l5\" (UniqueName: \"kubernetes.io/projected/a995cca6-542f-406e-a938-aa58715eac9a-kube-api-access-4q4l5\") pod \"kas-connection-checker-685895c7c9-48l5q\" (UID: \"a995cca6-542f-406e-a938-aa58715eac9a\") " pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618933 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e448fa5-814c-427f-854e-02708388f82a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98667931-42d6-42d2-b335-57b2d19d7dfc-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618963 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.618989 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619002 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619018 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e448fa5-814c-427f-854e-02708388f82a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.619032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619035 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619051 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfj4h\" (UniqueName: \"kubernetes.io/projected/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-kube-api-access-nfj4h\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619072 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-stats-auth\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-tmp\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619139 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619164 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-config-volume\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619193 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619223 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a59cf1-454d-4019-9e05-0108e38b8725-serving-cert\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619247 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619275 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-snapshots\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619322 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619339 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619354 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bbdae4d-8429-4f49-8d31-fe00df3d0701-cert\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.619558 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619376 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b764x\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619402 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b4895c8-b425-4009-a06b-47f1d51b8784-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619447 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxxd9\" (UniqueName: \"kubernetes.io/projected/02bb73c6-e6e2-4d77-8458-20ba459b831a-kube-api-access-qxxd9\") pod \"kas-connection-checker-685895c7c9-8bznz\" (UID: \"02bb73c6-e6e2-4d77-8458-20ba459b831a\") " pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619471 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-trusted-ca\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619495 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619543 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98667931-42d6-42d2-b335-57b2d19d7dfc-config\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619571 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619600 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9skt\" (UniqueName: \"kubernetes.io/projected/c7a59cf1-454d-4019-9e05-0108e38b8725-kube-api-access-b9skt\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619677 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619702 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-default-certificate\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619729 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82xf\" (UniqueName: \"kubernetes.io/projected/aac3b6b3-61bd-4ee2-b116-61fb0a786896-kube-api-access-l82xf\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619756 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/6b4895c8-b425-4009-a06b-47f1d51b8784-kube-api-access-8lcbr\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.620215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619775 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkmm\" (UniqueName: \"kubernetes.io/projected/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-kube-api-access-6pkmm\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5l82\" (UniqueName: \"kubernetes.io/projected/3bbdae4d-8429-4f49-8d31-fe00df3d0701-kube-api-access-j5l82\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bc681f0-27c6-4cad-8bf6-82835e009976-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619840 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6htdx\" (UniqueName: \"kubernetes.io/projected/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750-kube-api-access-6htdx\") pod \"kas-connection-checker-685895c7c9-k2tz8\" (UID: \"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750\") " pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619856 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-serving-cert\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619871 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rc9\" (UniqueName: \"kubernetes.io/projected/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4-kube-api-access-z2rc9\") pod \"network-check-source-6f78d59f99-vmfmg\" (UID: \"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619887 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-tmp-dir\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619930 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49dx\" (UniqueName: \"kubernetes.io/projected/98667931-42d6-42d2-b335-57b2d19d7dfc-kube-api-access-p49dx\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619964 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.620742 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.619979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.621364 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.621345 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sd2tb"] Nov 03 20:29:11.623401 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.623380 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-595485878c-mqdnk"] Nov 03 20:29:11.645961 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.645905 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.645961 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.645905 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.645961 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.645904 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.646180 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646161 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:29:11.646270 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646185 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.646317 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646272 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.646317 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646281 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.646447 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646436 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.646857 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646843 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.646937 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646882 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:29:11.646937 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646917 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.647029 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.646988 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.647029 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.647002 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.652499 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.652477 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:29:11.653115 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.653094 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:29:11.654468 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.654453 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.654559 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.654542 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.655384 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.655293 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:29:11.655490 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.655474 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.655555 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.655542 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.669043 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.669020 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.669109 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.669081 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.691717 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.691697 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:29:11.691717 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.691709 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.692065 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.692052 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:29:11.692168 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.692106 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:11.692753 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.692732 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:29:11.692808 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.692778 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:29:11.694608 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.694590 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:29:11.699032 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.699016 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:29:11.715719 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.715700 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:29:11.715836 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.715817 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:29:11.716156 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.716141 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:29:11.716156 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.716150 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:29:11.716314 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.716188 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:29:11.716867 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.716849 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2b69b"] Nov 03 20:29:11.717429 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.717415 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bw59n"] Nov 03 20:29:11.717995 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.717985 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-75cb4f9586-ps4d8"] Nov 03 20:29:11.718534 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.718511 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ljjts"] Nov 03 20:29:11.720580 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-trusted-ca\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.720658 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720589 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.720658 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720607 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.720658 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98667931-42d6-42d2-b335-57b2d19d7dfc-config\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.720765 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720741 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.720796 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9skt\" (UniqueName: \"kubernetes.io/projected/c7a59cf1-454d-4019-9e05-0108e38b8725-kube-api-access-b9skt\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.720796 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720791 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.720850 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720816 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.720850 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-default-certificate\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.720934 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82xf\" (UniqueName: \"kubernetes.io/projected/aac3b6b3-61bd-4ee2-b116-61fb0a786896-kube-api-access-l82xf\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.720934 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.720881 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/6b4895c8-b425-4009-a06b-47f1d51b8784-kube-api-access-8lcbr\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.721289 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721268 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkmm\" (UniqueName: \"kubernetes.io/projected/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-kube-api-access-6pkmm\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.721381 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721294 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98667931-42d6-42d2-b335-57b2d19d7dfc-config\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.721381 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721304 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5l82\" (UniqueName: \"kubernetes.io/projected/3bbdae4d-8429-4f49-8d31-fe00df3d0701-kube-api-access-j5l82\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.721381 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721357 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bc681f0-27c6-4cad-8bf6-82835e009976-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721388 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htdx\" (UniqueName: \"kubernetes.io/projected/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750-kube-api-access-6htdx\") pod \"kas-connection-checker-685895c7c9-k2tz8\" (UID: \"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750\") " pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-serving-cert\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721440 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rc9\" (UniqueName: \"kubernetes.io/projected/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4-kube-api-access-z2rc9\") pod \"network-check-source-6f78d59f99-vmfmg\" (UID: \"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721466 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721487 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-tmp-dir\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721511 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.721569 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49dx\" (UniqueName: \"kubernetes.io/projected/98667931-42d6-42d2-b335-57b2d19d7dfc-kube-api-access-p49dx\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.721927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721652 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.721927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721768 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-trusted-ca\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.721927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721780 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.721927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721798 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.721927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.721927 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721862 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-tmp-dir\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.722197 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.721975 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j877w\" (UniqueName: \"kubernetes.io/projected/5e448fa5-814c-427f-854e-02708388f82a-kube-api-access-j877w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.722197 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722077 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-config\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.722197 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722118 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.722197 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722145 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnlx\" (UniqueName: \"kubernetes.io/projected/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-kube-api-access-swnlx\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.722197 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722187 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.722442 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722200 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.722442 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722231 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bc681f0-27c6-4cad-8bf6-82835e009976-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:11.722442 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722375 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.722442 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722399 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d942d\" (UniqueName: \"kubernetes.io/projected/6ab99b87-827c-4aba-a47e-b86a847ca383-kube-api-access-d942d\") pod \"volume-data-source-validator-58c9f5f46d-b74gt\" (UID: \"6ab99b87-827c-4aba-a47e-b86a847ca383\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:11.722442 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722420 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q4l5\" (UniqueName: \"kubernetes.io/projected/a995cca6-542f-406e-a938-aa58715eac9a-kube-api-access-4q4l5\") pod \"kas-connection-checker-685895c7c9-48l5q\" (UID: \"a995cca6-542f-406e-a938-aa58715eac9a\") " pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722448 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e448fa5-814c-427f-854e-02708388f82a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98667931-42d6-42d2-b335-57b2d19d7dfc-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722500 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722581 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e448fa5-814c-427f-854e-02708388f82a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722657 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-config\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.722711 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722672 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj4h\" (UniqueName: \"kubernetes.io/projected/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-kube-api-access-nfj4h\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722807 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-stats-auth\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722840 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-tmp\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722866 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-config-volume\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722958 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a59cf1-454d-4019-9e05-0108e38b8725-serving-cert\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.722983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723013 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723037 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-snapshots\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723066 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.723127 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723106 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.723706 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723166 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.723706 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bbdae4d-8429-4f49-8d31-fe00df3d0701-cert\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.724016 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723845 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.724084 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724052 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e448fa5-814c-427f-854e-02708388f82a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.724131 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.723893 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.724255 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724234 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b764x\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.724342 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724320 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-tmp\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.724492 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724476 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b4895c8-b425-4009-a06b-47f1d51b8784-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.724556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724533 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.724675 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724575 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxd9\" (UniqueName: \"kubernetes.io/projected/02bb73c6-e6e2-4d77-8458-20ba459b831a-kube-api-access-qxxd9\") pod \"kas-connection-checker-685895c7c9-8bznz\" (UID: \"02bb73c6-e6e2-4d77-8458-20ba459b831a\") " pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:11.724780 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.724827 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.724674 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-snapshots\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.725592 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.725573 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.225558719 +0000 UTC m=+32.703090207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : configmap references non-existent config key: service-ca.crt Nov 03 20:29:11.725714 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.725691 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-config-volume\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.726608 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.726590 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-default-certificate\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.728088 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.728060 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-stats-auth\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.729019 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.728979 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.730463 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.730439 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b4895c8-b425-4009-a06b-47f1d51b8784-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.731960 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.731935 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e448fa5-814c-427f-854e-02708388f82a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.737972 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.737949 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:29:11.737972 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.737959 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:29:11.738083 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.737996 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:29:11.738083 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738010 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v9mbf\"" Nov 03 20:29:11.738853 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738837 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:29:11.738908 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738854 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:29:11.738908 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738867 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n968g\"" Nov 03 20:29:11.738908 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738891 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rrsvs\"" Nov 03 20:29:11.738908 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738905 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738917 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738907 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9nsh4\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738898 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738920 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gpslm\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738916 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdt7f\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738950 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738954 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mfn79\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738929 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738969 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zw8dl\"" Nov 03 20:29:11.739090 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.738973 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:29:11.739416 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.739104 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7sh8x\"" Nov 03 20:29:11.739416 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.739128 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:29:11.739416 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.739169 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-65mmw\"" Nov 03 20:29:11.739416 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.739241 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9fv67\"" Nov 03 20:29:11.739416 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.739253 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c56x6\"" Nov 03 20:29:11.739822 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.739809 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7"] Nov 03 20:29:11.740429 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.740415 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b"] Nov 03 20:29:11.740974 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.740957 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg"] Nov 03 20:29:11.741321 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.741308 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:11.741370 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.741322 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb4f9586-ps4d8: secret "image-registry-tls" not found Nov 03 20:29:11.741370 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.741356 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls podName:c85654a0-283c-4b2b-bc7c-253a053d9f4d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.24134489 +0000 UTC m=+32.718876355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls") pod "image-registry-75cb4f9586-ps4d8" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d") : secret "image-registry-tls" not found Nov 03 20:29:11.742225 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742210 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:11.742289 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742261 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls podName:e44f243f-ff0c-4e9a-9a91-b0079c2667b6 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.242250151 +0000 UTC m=+32.719781617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls") pod "dns-default-sd2tb" (UID: "e44f243f-ff0c-4e9a-9a91-b0079c2667b6") : secret "dns-default-metrics-tls" not found Nov 03 20:29:11.742289 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742276 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742295 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.242289776 +0000 UTC m=+32.719821241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : secret "router-metrics-certs-default" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742657 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742672 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595485878c-mqdnk: secret "image-registry-tls" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742709 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls podName:4382cc48-e16d-4e80-85af-ff5f5d715c7a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.242697619 +0000 UTC m=+32.720229087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls") pod "image-registry-595485878c-mqdnk" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a") : secret "image-registry-tls" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742737 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.742771 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert podName:5bc681f0-27c6-4cad-8bf6-82835e009976 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.242761197 +0000 UTC m=+32.720292662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2b69b" (UID: "5bc681f0-27c6-4cad-8bf6-82835e009976") : secret "networking-console-plugin-cert" not found Nov 03 20:29:11.743776 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.742818 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.744203 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.744166 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-serving-cert\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.744447 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.744430 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.744523 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.744508 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98667931-42d6-42d2-b335-57b2d19d7dfc-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.744933 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.744917 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:29:11.745031 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.744982 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls podName:aac3b6b3-61bd-4ee2-b116-61fb0a786896 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.244966668 +0000 UTC m=+32.722498145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqx9b" (UID: "aac3b6b3-61bd-4ee2-b116-61fb0a786896") : secret "samples-operator-tls" not found Nov 03 20:29:11.746227 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.746175 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:11.746366 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:11.746240 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls podName:6b4895c8-b425-4009-a06b-47f1d51b8784 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:12.2462247 +0000 UTC m=+32.723756179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-627kh" (UID: "6b4895c8-b425-4009-a06b-47f1d51b8784") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:11.746810 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.746791 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.746951 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.746931 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.748534 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.748517 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a59cf1-454d-4019-9e05-0108e38b8725-serving-cert\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.748762 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.748742 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bbdae4d-8429-4f49-8d31-fe00df3d0701-cert\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.786128 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.786101 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rc9\" (UniqueName: \"kubernetes.io/projected/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4-kube-api-access-z2rc9\") pod \"network-check-source-6f78d59f99-vmfmg\" (UID: \"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:11.786500 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.786478 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9skt\" (UniqueName: \"kubernetes.io/projected/c7a59cf1-454d-4019-9e05-0108e38b8725-kube-api-access-b9skt\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.786568 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.786545 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.786861 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.786842 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkmm\" (UniqueName: \"kubernetes.io/projected/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-kube-api-access-6pkmm\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:11.787031 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.787014 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5l82\" (UniqueName: \"kubernetes.io/projected/3bbdae4d-8429-4f49-8d31-fe00df3d0701-kube-api-access-j5l82\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:11.787335 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.787319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/6b4895c8-b425-4009-a06b-47f1d51b8784-kube-api-access-8lcbr\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:11.791534 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.791514 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49dx\" (UniqueName: \"kubernetes.io/projected/98667931-42d6-42d2-b335-57b2d19d7dfc-kube-api-access-p49dx\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.791648 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.791617 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:11.791743 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.791725 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j877w\" (UniqueName: \"kubernetes.io/projected/5e448fa5-814c-427f-854e-02708388f82a-kube-api-access-j877w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.791809 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.791789 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htdx\" (UniqueName: \"kubernetes.io/projected/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750-kube-api-access-6htdx\") pod \"kas-connection-checker-685895c7c9-k2tz8\" (UID: \"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750\") " pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:11.795600 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.795574 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnlx\" (UniqueName: \"kubernetes.io/projected/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-kube-api-access-swnlx\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.796438 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.796417 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d942d\" (UniqueName: \"kubernetes.io/projected/6ab99b87-827c-4aba-a47e-b86a847ca383-kube-api-access-d942d\") pod \"volume-data-source-validator-58c9f5f46d-b74gt\" (UID: \"6ab99b87-827c-4aba-a47e-b86a847ca383\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:11.796438 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.796426 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q4l5\" (UniqueName: \"kubernetes.io/projected/a995cca6-542f-406e-a938-aa58715eac9a-kube-api-access-4q4l5\") pod \"kas-connection-checker-685895c7c9-48l5q\" (UID: \"a995cca6-542f-406e-a938-aa58715eac9a\") " pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:11.796565 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.796458 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82xf\" (UniqueName: \"kubernetes.io/projected/aac3b6b3-61bd-4ee2-b116-61fb0a786896-kube-api-access-l82xf\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:11.798020 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.798000 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.798290 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.798271 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxd9\" (UniqueName: \"kubernetes.io/projected/02bb73c6-e6e2-4d77-8458-20ba459b831a-kube-api-access-qxxd9\") pod \"kas-connection-checker-685895c7c9-8bznz\" (UID: \"02bb73c6-e6e2-4d77-8458-20ba459b831a\") " pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:11.800463 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.800445 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj4h\" (UniqueName: \"kubernetes.io/projected/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-kube-api-access-nfj4h\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:11.800744 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.800728 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b764x\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:11.812041 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.812025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:11.818163 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.818147 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:11.823699 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.823676 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:11.830097 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.830033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:11.835981 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.835960 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:11.848294 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.848231 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:11.864669 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.860639 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:11.890786 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.890713 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:11.900766 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.900681 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:11.924789 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:11.916342 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:12.220614 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.220550 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt"] Nov 03 20:29:12.225555 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.225413 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-k2tz8"] Nov 03 20:29:12.227919 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.227894 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab99b87_827c_4aba_a47e_b86a847ca383.slice/crio-2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881 WatchSource:0}: Error finding container 2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881: Status 404 returned error can't find the container with id 2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881 Nov 03 20:29:12.230977 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.230956 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:12.231148 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.231117 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.231102174 +0000 UTC m=+33.708633644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : configmap references non-existent config key: service-ca.crt Nov 03 20:29:12.231686 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.231664 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6521fc8f_d7c8_46cb_b50c_eb7ff5ec1750.slice/crio-87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50 WatchSource:0}: Error finding container 87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50: Status 404 returned error can't find the container with id 87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50 Nov 03 20:29:12.239392 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.239372 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-48l5q"] Nov 03 20:29:12.249285 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.249255 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda995cca6_542f_406e_a938_aa58715eac9a.slice/crio-75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3 WatchSource:0}: Error finding container 75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3: Status 404 returned error can't find the container with id 75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3 Nov 03 20:29:12.271339 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.271274 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8bznz"] Nov 03 20:29:12.275843 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.275753 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7"] Nov 03 20:29:12.275843 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.275802 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-ljjts"] Nov 03 20:29:12.280218 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.280193 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bb73c6_e6e2_4d77_8458_20ba459b831a.slice/crio-157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5 WatchSource:0}: Error finding container 157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5: Status 404 returned error can't find the container with id 157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5 Nov 03 20:29:12.281322 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.281300 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a59cf1_454d_4019_9e05_0108e38b8725.slice/crio-f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28 WatchSource:0}: Error finding container f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28: Status 404 returned error can't find the container with id f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28 Nov 03 20:29:12.299278 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.299244 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8bznz" event={"ID":"02bb73c6-e6e2-4d77-8458-20ba459b831a","Type":"ContainerStarted","Data":"157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5"} Nov 03 20:29:12.301091 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.301029 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" event={"ID":"98667931-42d6-42d2-b335-57b2d19d7dfc","Type":"ContainerStarted","Data":"093b3fa970ad305da00cbb434365c1b3bfbe2da80a8faaf908a1434b2f88a1f5"} Nov 03 20:29:12.303824 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.303762 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" event={"ID":"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750","Type":"ContainerStarted","Data":"87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50"} Nov 03 20:29:12.305336 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.305278 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-48l5q" event={"ID":"a995cca6-542f-406e-a938-aa58715eac9a","Type":"ContainerStarted","Data":"75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3"} Nov 03 20:29:12.306991 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.306930 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" event={"ID":"c7a59cf1-454d-4019-9e05-0108e38b8725","Type":"ContainerStarted","Data":"f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28"} Nov 03 20:29:12.308713 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.308660 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" event={"ID":"6ab99b87-827c-4aba-a47e-b86a847ca383","Type":"ContainerStarted","Data":"2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881"} Nov 03 20:29:12.325946 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.325913 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-zd9mv"] Nov 03 20:29:12.331731 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331704 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:12.331834 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:12.331888 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:12.331888 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:12.331992 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:12.331992 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331970 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:12.332100 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.331995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:12.332155 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332146 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:12.332205 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332159 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595485878c-mqdnk: secret "image-registry-tls" not found Nov 03 20:29:12.332255 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332206 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls podName:4382cc48-e16d-4e80-85af-ff5f5d715c7a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332189907 +0000 UTC m=+33.809721390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls") pod "image-registry-595485878c-mqdnk" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a") : secret "image-registry-tls" not found Nov 03 20:29:12.332384 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332368 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:29:12.332460 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332419 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert podName:5bc681f0-27c6-4cad-8bf6-82835e009976 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332404585 +0000 UTC m=+33.809936054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2b69b" (UID: "5bc681f0-27c6-4cad-8bf6-82835e009976") : secret "networking-console-plugin-cert" not found Nov 03 20:29:12.332515 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332484 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:29:12.332564 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332520 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls podName:aac3b6b3-61bd-4ee2-b116-61fb0a786896 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332510048 +0000 UTC m=+33.810041528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqx9b" (UID: "aac3b6b3-61bd-4ee2-b116-61fb0a786896") : secret "samples-operator-tls" not found Nov 03 20:29:12.332617 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332572 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:12.332617 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332603 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls podName:6b4895c8-b425-4009-a06b-47f1d51b8784 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332593376 +0000 UTC m=+33.810124854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-627kh" (UID: "6b4895c8-b425-4009-a06b-47f1d51b8784") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:12.332617 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332678 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:12.332617 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332689 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb4f9586-ps4d8: secret "image-registry-tls" not found Nov 03 20:29:12.332907 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332720 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls podName:c85654a0-283c-4b2b-bc7c-253a053d9f4d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332709531 +0000 UTC m=+33.810240997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls") pod "image-registry-75cb4f9586-ps4d8" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d") : secret "image-registry-tls" not found Nov 03 20:29:12.332907 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332773 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:12.332907 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332803 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls podName:e44f243f-ff0c-4e9a-9a91-b0079c2667b6 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332793893 +0000 UTC m=+33.810325371 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls") pod "dns-default-sd2tb" (UID: "e44f243f-ff0c-4e9a-9a91-b0079c2667b6") : secret "dns-default-metrics-tls" not found Nov 03 20:29:12.332907 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332860 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:29:12.332907 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.332891 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:13.332881743 +0000 UTC m=+33.810413222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : secret "router-metrics-certs-default" not found Nov 03 20:29:12.339040 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.339013 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea833ecb_ee75_45fd_b689_2b7c0c9a711c.slice/crio-938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88 WatchSource:0}: Error finding container 938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88: Status 404 returned error can't find the container with id 938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88 Nov 03 20:29:12.364174 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.364153 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg"] Nov 03 20:29:12.369234 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.369125 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bw59n"] Nov 03 20:29:12.370426 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.370404 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb60bf0ba_cf9a_4238_bfb2_4b13de61d5b4.slice/crio-3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3 WatchSource:0}: Error finding container 3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3: Status 404 returned error can't find the container with id 3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3 Nov 03 20:29:12.384460 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.384438 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bbdae4d_8429_4f49_8d31_fe00df3d0701.slice/crio-07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3 WatchSource:0}: Error finding container 07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3: Status 404 returned error can't find the container with id 07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3 Nov 03 20:29:12.484724 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.484649 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7"] Nov 03 20:29:12.492305 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:12.492286 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e448fa5_814c_427f_854e_02708388f82a.slice/crio-a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a WatchSource:0}: Error finding container a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a: Status 404 returned error can't find the container with id a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a Nov 03 20:29:12.936953 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.936927 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:12.937086 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:12.936980 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:12.937086 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.937057 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 20:29:12.937176 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.937056 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:29:12.937176 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.937100 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret podName:c947407f-1cc6-4c01-997e-99e19a3c139c nodeName:}" failed. No retries permitted until 2026-11-03 20:29:44.937089243 +0000 UTC m=+65.414620741 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret") pod "global-pull-secret-syncer-5khcq" (UID: "c947407f-1cc6-4c01-997e-99e19a3c139c") : object "kube-system"/"original-pull-secret" not registered Nov 03 20:29:12.937176 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:12.937135 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs podName:ea25f895-6d8c-4bf1-abd4-6e29c4a772e7 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:44.937122257 +0000 UTC m=+65.414653722 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs") pod "network-metrics-daemon-vfs2r" (UID: "ea25f895-6d8c-4bf1-abd4-6e29c4a772e7") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 20:29:13.038139 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.038118 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:13.049960 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.049939 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:13.093754 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.093730 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:13.093863 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.093761 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:13.093863 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.093857 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:13.143125 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.143105 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:29:13.143228 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.143202 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\"" Nov 03 20:29:13.143228 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.143208 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\"" Nov 03 20:29:13.143305 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.143225 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\"" Nov 03 20:29:13.143305 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.143211 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:29:13.161343 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.161321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:13.240176 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.240147 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:13.240494 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.240256 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.240240514 +0000 UTC m=+35.717771979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : configmap references non-existent config key: service-ca.crt Nov 03 20:29:13.313272 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.313203 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" event={"ID":"5e448fa5-814c-427f-854e-02708388f82a","Type":"ContainerStarted","Data":"a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a"} Nov 03 20:29:13.315895 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.315865 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8bznz" event={"ID":"02bb73c6-e6e2-4d77-8458-20ba459b831a","Type":"ContainerStarted","Data":"6739942a1bfbdefda8b3fcc1f45d5dce9eb0984f63641e9ed96b95a74234751e"} Nov 03 20:29:13.318376 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.318352 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bw59n" event={"ID":"3bbdae4d-8429-4f49-8d31-fe00df3d0701","Type":"ContainerStarted","Data":"07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3"} Nov 03 20:29:13.320043 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.320000 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" event={"ID":"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4","Type":"ContainerStarted","Data":"3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3"} Nov 03 20:29:13.322233 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.322187 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" event={"ID":"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750","Type":"ContainerStarted","Data":"a00f3b884c587ba4aecae28b1e7598dccc13d1179ce5cd1ca4f714d9ac06eb2b"} Nov 03 20:29:13.325247 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.325188 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" event={"ID":"ea833ecb-ee75-45fd-b689-2b7c0c9a711c","Type":"ContainerStarted","Data":"938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88"} Nov 03 20:29:13.331913 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.331520 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-48l5q" event={"ID":"a995cca6-542f-406e-a938-aa58715eac9a","Type":"ContainerStarted","Data":"32e5787621a4021a54f2e6c531f1c9f7392b1f267892bc3de634fa50dd26876b"} Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341501 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341554 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341683 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341753 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.341804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.341907 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.341954 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls podName:aac3b6b3-61bd-4ee2-b116-61fb0a786896 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.341938892 +0000 UTC m=+35.819470357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqx9b" (UID: "aac3b6b3-61bd-4ee2-b116-61fb0a786896") : secret "samples-operator-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342024 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342057 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls podName:6b4895c8-b425-4009-a06b-47f1d51b8784 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.342046755 +0000 UTC m=+35.819578225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-627kh" (UID: "6b4895c8-b425-4009-a06b-47f1d51b8784") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342118 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342128 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb4f9586-ps4d8: secret "image-registry-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342158 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls podName:c85654a0-283c-4b2b-bc7c-253a053d9f4d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.342148662 +0000 UTC m=+35.819680129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls") pod "image-registry-75cb4f9586-ps4d8" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d") : secret "image-registry-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342226 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:13.342480 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342248 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls podName:e44f243f-ff0c-4e9a-9a91-b0079c2667b6 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.342241861 +0000 UTC m=+35.819773330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls") pod "dns-default-sd2tb" (UID: "e44f243f-ff0c-4e9a-9a91-b0079c2667b6") : secret "dns-default-metrics-tls" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342291 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342312 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.342306652 +0000 UTC m=+35.819838120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : secret "router-metrics-certs-default" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342355 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342362 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595485878c-mqdnk: secret "image-registry-tls" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342382 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls podName:4382cc48-e16d-4e80-85af-ff5f5d715c7a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.34237469 +0000 UTC m=+35.819906159 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls") pod "image-registry-595485878c-mqdnk" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a") : secret "image-registry-tls" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342416 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:29:13.343156 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:13.342434 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert podName:5bc681f0-27c6-4cad-8bf6-82835e009976 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:15.342428577 +0000 UTC m=+35.819960047 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2b69b" (UID: "5bc681f0-27c6-4cad-8bf6-82835e009976") : secret "networking-console-plugin-cert" not found Nov 03 20:29:13.515709 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.515598 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8bznz" podStartSLOduration=421.515584984 podStartE2EDuration="7m1.515584984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:13.514145635 +0000 UTC m=+33.991677123" watchObservedRunningTime="2026-11-03 20:29:13.515584984 +0000 UTC m=+33.993116659" Nov 03 20:29:13.542246 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.542210 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9x8q2"] Nov 03 20:29:13.548747 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:13.548717 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe95bd0d_a5b5_44db_97ae_d75a6860a71a.slice/crio-d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035 WatchSource:0}: Error finding container d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035: Status 404 returned error can't find the container with id d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035 Nov 03 20:29:13.623328 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.623250 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" podStartSLOduration=421.623236474 podStartE2EDuration="7m1.623236474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:13.622566751 +0000 UTC m=+34.100098240" watchObservedRunningTime="2026-11-03 20:29:13.623236474 +0000 UTC m=+34.100767938" Nov 03 20:29:13.733562 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:13.733313 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-48l5q" podStartSLOduration=421.73329589 podStartE2EDuration="7m1.73329589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:13.732638574 +0000 UTC m=+34.210170056" watchObservedRunningTime="2026-11-03 20:29:13.73329589 +0000 UTC m=+34.210827378" Nov 03 20:29:14.341845 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:14.341806 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9x8q2" event={"ID":"fe95bd0d-a5b5-44db-97ae-d75a6860a71a","Type":"ContainerStarted","Data":"d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035"} Nov 03 20:29:15.265148 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.264472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:15.265148 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.264764 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.264745607 +0000 UTC m=+39.742277075 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : configmap references non-existent config key: service-ca.crt Nov 03 20:29:15.350037 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.349991 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" event={"ID":"6ab99b87-827c-4aba-a47e-b86a847ca383","Type":"ContainerStarted","Data":"4e73a4d93cbec9d268f9b3c9d9a0dd2bb40e87e817eee3adbf08f7f14389ce16"} Nov 03 20:29:15.365574 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:15.365574 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365569 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:15.365785 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365637 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:15.365785 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:15.365785 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:15.365785 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365756 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:15.365785 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.365808 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:15.366104 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.365945 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:29:15.366104 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366031 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls podName:aac3b6b3-61bd-4ee2-b116-61fb0a786896 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366013758 +0000 UTC m=+39.843545230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqx9b" (UID: "aac3b6b3-61bd-4ee2-b116-61fb0a786896") : secret "samples-operator-tls" not found Nov 03 20:29:15.366422 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366405 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:15.366502 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366458 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls podName:6b4895c8-b425-4009-a06b-47f1d51b8784 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366444461 +0000 UTC m=+39.843975937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-627kh" (UID: "6b4895c8-b425-4009-a06b-47f1d51b8784") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:15.366616 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366530 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:15.366616 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366541 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb4f9586-ps4d8: secret "image-registry-tls" not found Nov 03 20:29:15.366616 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366573 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls podName:c85654a0-283c-4b2b-bc7c-253a053d9f4d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366561941 +0000 UTC m=+39.844093411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls") pod "image-registry-75cb4f9586-ps4d8" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d") : secret "image-registry-tls" not found Nov 03 20:29:15.366799 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366642 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:15.366799 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366688 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls podName:e44f243f-ff0c-4e9a-9a91-b0079c2667b6 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366677166 +0000 UTC m=+39.844208643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls") pod "dns-default-sd2tb" (UID: "e44f243f-ff0c-4e9a-9a91-b0079c2667b6") : secret "dns-default-metrics-tls" not found Nov 03 20:29:15.366799 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366740 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:29:15.366799 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366770 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366760214 +0000 UTC m=+39.844291683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : secret "router-metrics-certs-default" not found Nov 03 20:29:15.367048 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366834 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:15.367048 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366847 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595485878c-mqdnk: secret "image-registry-tls" not found Nov 03 20:29:15.367048 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366881 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls podName:4382cc48-e16d-4e80-85af-ff5f5d715c7a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366870151 +0000 UTC m=+39.844401627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls") pod "image-registry-595485878c-mqdnk" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a") : secret "image-registry-tls" not found Nov 03 20:29:15.367048 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366939 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:29:15.367048 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:15.366969 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert podName:5bc681f0-27c6-4cad-8bf6-82835e009976 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:19.366959459 +0000 UTC m=+39.844490939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2b69b" (UID: "5bc681f0-27c6-4cad-8bf6-82835e009976") : secret "networking-console-plugin-cert" not found Nov 03 20:29:15.460486 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:15.459690 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" podStartSLOduration=369.827264881 podStartE2EDuration="6m12.459676434s" totalImagesPullingTime="2.632411553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:15.458600119 +0000 UTC m=+35.936131607" watchObservedRunningTime="2026-11-03 20:29:15.459676434 +0000 UTC m=+35.937207921" Nov 03 20:29:17.360019 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:17.359911 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" event={"ID":"98667931-42d6-42d2-b335-57b2d19d7dfc","Type":"ContainerStarted","Data":"a6215f8eba2cccd47d7cd2bf203504506ac431492422a68707a7b4ac1f2649b5"} Nov 03 20:29:17.362573 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:17.361951 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/0.log" Nov 03 20:29:17.362573 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:17.361991 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7a59cf1-454d-4019-9e05-0108e38b8725" containerID="3713201aa607ad66db559821c45434dbd44a6677b547e297e3493433dc29a53b" exitCode=255 Nov 03 20:29:17.362573 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:17.362027 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" event={"ID":"c7a59cf1-454d-4019-9e05-0108e38b8725","Type":"ContainerDied","Data":"3713201aa607ad66db559821c45434dbd44a6677b547e297e3493433dc29a53b"} Nov 03 20:29:17.362573 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:17.362316 2572 scope.go:122] "RemoveContainer" containerID="3713201aa607ad66db559821c45434dbd44a6677b547e297e3493433dc29a53b" Nov 03 20:29:17.470868 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:17.470812 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" podStartSLOduration=366.741447967 podStartE2EDuration="6m11.470795687s" totalImagesPullingTime="4.72934772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:17.469753729 +0000 UTC m=+37.947285216" watchObservedRunningTime="2026-11-03 20:29:17.470795687 +0000 UTC m=+37.948327174" Nov 03 20:29:18.369411 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.369328 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:29:18.369855 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.369829 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/0.log" Nov 03 20:29:18.369914 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.369867 2572 generic.go:356] "Generic (PLEG): container finished" podID="c7a59cf1-454d-4019-9e05-0108e38b8725" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" exitCode=255 Nov 03 20:29:18.370731 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.369976 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" event={"ID":"c7a59cf1-454d-4019-9e05-0108e38b8725","Type":"ContainerDied","Data":"f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e"} Nov 03 20:29:18.370731 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.370038 2572 scope.go:122] "RemoveContainer" containerID="3713201aa607ad66db559821c45434dbd44a6677b547e297e3493433dc29a53b" Nov 03 20:29:18.370731 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.370310 2572 scope.go:122] "RemoveContainer" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" Nov 03 20:29:18.370731 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:18.370502 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ljjts_openshift-console-operator(c7a59cf1-454d-4019-9e05-0108e38b8725)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" podUID="c7a59cf1-454d-4019-9e05-0108e38b8725" Nov 03 20:29:18.373553 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:18.373465 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" event={"ID":"ea833ecb-ee75-45fd-b689-2b7c0c9a711c","Type":"ContainerStarted","Data":"532206674ebd8f316be3a2000f31ee2e745eb7fd7d8ad83f2b546bdf9109d3d4"} Nov 03 20:29:19.294928 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.294879 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" podStartSLOduration=365.844181749 podStartE2EDuration="6m11.294863545s" totalImagesPullingTime="5.450681796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:18.600064018 +0000 UTC m=+39.077595506" watchObservedRunningTime="2026-11-03 20:29:19.294863545 +0000 UTC m=+39.772395032" Nov 03 20:29:19.295451 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.295435 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dpsrk"] Nov 03 20:29:19.308598 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.308572 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:19.308756 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.308739 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.308723253 +0000 UTC m=+47.786254723 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : configmap references non-existent config key: service-ca.crt Nov 03 20:29:19.309181 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.309167 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.357945 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.357918 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dpsrk"] Nov 03 20:29:19.358325 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.358308 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:29:19.377965 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.377944 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:29:19.378327 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.378233 2572 scope.go:122] "RemoveContainer" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" Nov 03 20:29:19.378427 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.378402 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ljjts_openshift-console-operator(c7a59cf1-454d-4019-9e05-0108e38b8725)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" podUID="c7a59cf1-454d-4019-9e05-0108e38b8725" Nov 03 20:29:19.379670 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.379643 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" event={"ID":"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4","Type":"ContainerStarted","Data":"61b23e862c744dc469c1238416aca83cbb6ea0c90c3508870f0bc8e4473ac72d"} Nov 03 20:29:19.402708 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.402686 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:29:19.403031 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.403012 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9t9mk\"" Nov 03 20:29:19.409417 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409396 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52568b14-d428-42c1-af42-2e15763fc1b3-data-volume\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.409492 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409437 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:19.409492 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409466 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpv4\" (UniqueName: \"kubernetes.io/projected/52568b14-d428-42c1-af42-2e15763fc1b3-kube-api-access-smpv4\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.409583 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:19.409583 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409564 2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 20:29:19.409691 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409584 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:19.409691 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409614 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls podName:e44f243f-ff0c-4e9a-9a91-b0079c2667b6 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.409601851 +0000 UTC m=+47.887133316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls") pod "dns-default-sd2tb" (UID: "e44f243f-ff0c-4e9a-9a91-b0079c2667b6") : secret "dns-default-metrics-tls" not found Nov 03 20:29:19.409794 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409700 2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 20:29:19.409794 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409744 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:19.409794 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409757 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.409739636 +0000 UTC m=+47.887271119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : secret "router-metrics-certs-default" not found Nov 03 20:29:19.409794 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409763 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-595485878c-mqdnk: secret "image-registry-tls" not found Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409812 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls podName:4382cc48-e16d-4e80-85af-ff5f5d715c7a nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.40979617 +0000 UTC m=+47.887327648 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls") pod "image-registry-595485878c-mqdnk" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a") : secret "image-registry-tls" not found Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409814 2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409858 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert podName:5bc681f0-27c6-4cad-8bf6-82835e009976 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.409844618 +0000 UTC m=+47.887376091 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-2b69b" (UID: "5bc681f0-27c6-4cad-8bf6-82835e009976") : secret "networking-console-plugin-cert" not found Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409876 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52568b14-d428-42c1-af42-2e15763fc1b3-crio-socket\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.409925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.409997 2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 20:29:19.410023 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.410008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.410045 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls podName:aac3b6b3-61bd-4ee2-b116-61fb0a786896 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.410032208 +0000 UTC m=+47.887563676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vqx9b" (UID: "aac3b6b3-61bd-4ee2-b116-61fb0a786896") : secret "samples-operator-tls" not found Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.410078 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.410079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.410109 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls podName:6b4895c8-b425-4009-a06b-47f1d51b8784 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.410097779 +0000 UTC m=+47.887629250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-627kh" (UID: "6b4895c8-b425-4009-a06b-47f1d51b8784") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.410140 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") pod \"image-registry-75cb4f9586-ps4d8\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.410168 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.410251 2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.410264 2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-75cb4f9586-ps4d8: secret "image-registry-tls" not found Nov 03 20:29:19.410317 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.410314 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls podName:c85654a0-283c-4b2b-bc7c-253a053d9f4d nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.410302266 +0000 UTC m=+47.887833751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls") pod "image-registry-75cb4f9586-ps4d8" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d") : secret "image-registry-tls" not found Nov 03 20:29:19.511380 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.511356 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.511476 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.511432 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.511978 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52568b14-d428-42c1-af42-2e15763fc1b3-data-volume\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.512137 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.512209 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:20.012186221 +0000 UTC m=+40.489717687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.512252 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.512265 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpv4\" (UniqueName: \"kubernetes.io/projected/52568b14-d428-42c1-af42-2e15763fc1b3-kube-api-access-smpv4\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.512289 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52568b14-d428-42c1-af42-2e15763fc1b3-data-volume\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.512497 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:19.512330 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:20.012305618 +0000 UTC m=+40.489837097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : configmap "kube-rbac-proxy" not found Nov 03 20:29:19.515329 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.512612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52568b14-d428-42c1-af42-2e15763fc1b3-crio-socket\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.515329 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.513162 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52568b14-d428-42c1-af42-2e15763fc1b3-crio-socket\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.527978 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.527924 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" podStartSLOduration=367.20333754 podStartE2EDuration="6m13.527908117s" totalImagesPullingTime="6.324570577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:19.526849226 +0000 UTC m=+40.004380713" watchObservedRunningTime="2026-11-03 20:29:19.527908117 +0000 UTC m=+40.005439609" Nov 03 20:29:19.533029 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.533009 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:29:19.545746 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.545688 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.572977 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.572953 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpv4\" (UniqueName: \"kubernetes.io/projected/52568b14-d428-42c1-af42-2e15763fc1b3-kube-api-access-smpv4\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:19.589923 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.589896 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:29:19.714027 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.713995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.714169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.714132 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.714214 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.714177 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.714309 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.714280 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.815362 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.815337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.815473 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.815404 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.815473 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.815436 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.815582 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.815485 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.815835 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.815811 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.815959 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.815943 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.816187 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.816169 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.873445 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.873412 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.881213 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.881184 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:19.972671 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:19.972502 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x5fbx"] Nov 03 20:29:20.006115 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.006083 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.016635 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.016596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:20.016754 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.016664 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:20.016754 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:20.016743 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:20.016838 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:20.016793 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:21.016779458 +0000 UTC m=+41.494310923 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:20.016838 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:20.016795 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:29:20.016921 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:20.016861 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:21.016843806 +0000 UTC m=+41.494375275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : configmap "kube-rbac-proxy" not found Nov 03 20:29:20.029192 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.029169 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x5fbx"] Nov 03 20:29:20.056454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.055482 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:29:20.056454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.055517 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:29:20.056454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.055640 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-245pv\"" Nov 03 20:29:20.056454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.055709 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:29:20.056454 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.055764 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:29:20.117811 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.117741 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmm8\" (UniqueName: \"kubernetes.io/projected/3086db43-8d5a-4196-ab9e-8c7f7832c232-kube-api-access-jwmm8\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.117811 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.117799 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-key\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.117960 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.117925 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-cabundle\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.219166 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.219132 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmm8\" (UniqueName: \"kubernetes.io/projected/3086db43-8d5a-4196-ab9e-8c7f7832c232-kube-api-access-jwmm8\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.219308 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.219193 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-key\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.219356 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.219305 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-cabundle\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.220131 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.220109 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-cabundle\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.221585 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.221556 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-key\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.271643 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.271609 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:29:20.277610 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:20.277584 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ab6c3f_8d7c_440d_a7d8_4917dc10f26a.slice/crio-ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e WatchSource:0}: Error finding container ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e: Status 404 returned error can't find the container with id ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e Nov 03 20:29:20.278302 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.278285 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmm8\" (UniqueName: \"kubernetes.io/projected/3086db43-8d5a-4196-ab9e-8c7f7832c232-kube-api-access-jwmm8\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.330531 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.330499 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:20.386081 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.386050 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bw59n" event={"ID":"3bbdae4d-8429-4f49-8d31-fe00df3d0701","Type":"ContainerStarted","Data":"c6145e0be107d390ced69566b25c07c55cf0e48951e0567b6bb2c74afda04423"} Nov 03 20:29:20.387926 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.387896 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" event={"ID":"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a","Type":"ContainerStarted","Data":"d21b2270103421f77f86dd2df622b214d341e6180f52113e366fd567a8685649"} Nov 03 20:29:20.388029 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.387949 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" event={"ID":"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a","Type":"ContainerStarted","Data":"ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e"} Nov 03 20:29:20.389849 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.389826 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" event={"ID":"5e448fa5-814c-427f-854e-02708388f82a","Type":"ContainerStarted","Data":"9f28042b548d8cf83c78c57a070d3353b6252c7cafc0d076fd4a886c289c907f"} Nov 03 20:29:20.391661 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.391469 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9x8q2" event={"ID":"fe95bd0d-a5b5-44db-97ae-d75a6860a71a","Type":"ContainerStarted","Data":"92485c919e6b4452869dbefcdd92dda60262f159219ccd04233ab840ea099377"} Nov 03 20:29:20.496362 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.496313 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bw59n" podStartSLOduration=2.155290377 podStartE2EDuration="9.496299543s" totalImagesPullingTime="7.341009166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:20.495174635 +0000 UTC m=+40.972706122" watchObservedRunningTime="2026-11-03 20:29:20.496299543 +0000 UTC m=+40.973831029" Nov 03 20:29:20.604087 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.604042 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" podStartSLOduration=1.604029826 podStartE2EDuration="1.604029826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:20.603353604 +0000 UTC m=+41.080885090" watchObservedRunningTime="2026-11-03 20:29:20.604029826 +0000 UTC m=+41.081561313" Nov 03 20:29:20.713458 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.713401 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" podStartSLOduration=364.768244542 podStartE2EDuration="6m11.713386811s" totalImagesPullingTime="6.945142269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:20.712287058 +0000 UTC m=+41.189818545" watchObservedRunningTime="2026-11-03 20:29:20.713386811 +0000 UTC m=+41.190918295" Nov 03 20:29:20.734425 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.734398 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-x5fbx"] Nov 03 20:29:20.740954 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:20.740928 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3086db43_8d5a_4196_ab9e_8c7f7832c232.slice/crio-f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9 WatchSource:0}: Error finding container f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9: Status 404 returned error can't find the container with id f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9 Nov 03 20:29:20.824533 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:20.824466 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9x8q2" podStartSLOduration=33.920584235 podStartE2EDuration="39.8244417s" totalImagesPullingTime="5.903857465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:20.823419379 +0000 UTC m=+41.300950869" watchObservedRunningTime="2026-11-03 20:29:20.8244417 +0000 UTC m=+41.301973189" Nov 03 20:29:21.026321 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.026251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:21.026471 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:21.026364 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:29:21.026471 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:21.026442 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:23.026411012 +0000 UTC m=+43.503942497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : configmap "kube-rbac-proxy" not found Nov 03 20:29:21.026588 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.026573 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:21.026727 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:21.026715 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:21.026784 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:21.026765 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:23.026750828 +0000 UTC m=+43.504282318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:21.367642 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.367557 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d"] Nov 03 20:29:21.393049 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.393019 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh"] Nov 03 20:29:21.393499 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.393159 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:21.409157 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.409134 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" event={"ID":"3086db43-8d5a-4196-ab9e-8c7f7832c232","Type":"ContainerStarted","Data":"af6ece119eda9209257f04180d04540e85031acacf332d4fa50630132e9903dd"} Nov 03 20:29:21.409271 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.409161 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" event={"ID":"3086db43-8d5a-4196-ab9e-8c7f7832c232","Type":"ContainerStarted","Data":"f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9"} Nov 03 20:29:21.409271 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.409180 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:21.409271 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.409244 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:21.424468 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.424445 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d"] Nov 03 20:29:21.442252 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.442229 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:21.442393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.442282 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mx84v\"" Nov 03 20:29:21.442393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.442315 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:29:21.450941 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.450912 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh"] Nov 03 20:29:21.532382 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.532352 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvqhp\" (UniqueName: \"kubernetes.io/projected/b91b91f6-e725-4d7b-b472-9e76650979ab-kube-api-access-kvqhp\") pod \"migrator-66df5d496b-977vh\" (UID: \"b91b91f6-e725-4d7b-b472-9e76650979ab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:21.532716 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.532693 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv629\" (UniqueName: \"kubernetes.io/projected/ff18aebf-b9af-4296-aaad-5d24fd9304ea-kube-api-access-kv629\") pod \"migrator-66df5d496b-rn27d\" (UID: \"ff18aebf-b9af-4296-aaad-5d24fd9304ea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:21.633825 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.633793 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqhp\" (UniqueName: \"kubernetes.io/projected/b91b91f6-e725-4d7b-b472-9e76650979ab-kube-api-access-kvqhp\") pod \"migrator-66df5d496b-977vh\" (UID: \"b91b91f6-e725-4d7b-b472-9e76650979ab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:21.634007 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.633989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv629\" (UniqueName: \"kubernetes.io/projected/ff18aebf-b9af-4296-aaad-5d24fd9304ea-kube-api-access-kv629\") pod \"migrator-66df5d496b-rn27d\" (UID: \"ff18aebf-b9af-4296-aaad-5d24fd9304ea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:21.691247 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.691219 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqhp\" (UniqueName: \"kubernetes.io/projected/b91b91f6-e725-4d7b-b472-9e76650979ab-kube-api-access-kvqhp\") pod \"migrator-66df5d496b-977vh\" (UID: \"b91b91f6-e725-4d7b-b472-9e76650979ab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:21.691352 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.691319 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv629\" (UniqueName: \"kubernetes.io/projected/ff18aebf-b9af-4296-aaad-5d24fd9304ea-kube-api-access-kv629\") pod \"migrator-66df5d496b-rn27d\" (UID: \"ff18aebf-b9af-4296-aaad-5d24fd9304ea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:21.716188 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.716166 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:21.718650 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.718582 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" podStartSLOduration=2.718567914 podStartE2EDuration="2.718567914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:21.717139695 +0000 UTC m=+42.194671183" watchObservedRunningTime="2026-11-03 20:29:21.718567914 +0000 UTC m=+42.196099402" Nov 03 20:29:21.732294 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.732273 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:21.837729 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.837694 2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:21.838189 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.838154 2572 scope.go:122] "RemoveContainer" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" Nov 03 20:29:21.838386 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:21.838363 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ljjts_openshift-console-operator(c7a59cf1-454d-4019-9e05-0108e38b8725)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" podUID="c7a59cf1-454d-4019-9e05-0108e38b8725" Nov 03 20:29:21.842307 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:21.842267 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:22.110866 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.109125 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d"] Nov 03 20:29:22.122767 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:22.122730 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff18aebf_b9af_4296_aaad_5d24fd9304ea.slice/crio-f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d WatchSource:0}: Error finding container f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d: Status 404 returned error can't find the container with id f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d Nov 03 20:29:22.164781 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.164751 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh"] Nov 03 20:29:22.171275 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:22.171253 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91b91f6_e725_4d7b_b472_9e76650979ab.slice/crio-ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97 WatchSource:0}: Error finding container ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97: Status 404 returned error can't find the container with id ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97 Nov 03 20:29:22.402328 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.402290 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" event={"ID":"b91b91f6-e725-4d7b-b472-9e76650979ab","Type":"ContainerStarted","Data":"ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97"} Nov 03 20:29:22.403605 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.403571 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" event={"ID":"ff18aebf-b9af-4296-aaad-5d24fd9304ea","Type":"ContainerStarted","Data":"f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d"} Nov 03 20:29:22.403892 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.403867 2572 scope.go:122] "RemoveContainer" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" Nov 03 20:29:22.404160 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:22.404141 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-ljjts_openshift-console-operator(c7a59cf1-454d-4019-9e05-0108e38b8725)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" podUID="c7a59cf1-454d-4019-9e05-0108e38b8725" Nov 03 20:29:22.575190 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.575157 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75cb4f9586-ps4d8"] Nov 03 20:29:22.575352 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:22.575334 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" podUID="c85654a0-283c-4b2b-bc7c-253a053d9f4d" Nov 03 20:29:22.625365 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.625336 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b5c54f4d6-pwkct"] Nov 03 20:29:22.652115 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.652092 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.691880 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.691814 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5c54f4d6-pwkct"] Nov 03 20:29:22.749050 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749027 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749170 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749170 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749103 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749236 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749188 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749236 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749214 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749295 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749277 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749358 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749345 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.749421 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.749408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850240 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850211 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850240 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850398 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850277 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850442 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850420 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850556 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850535 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850702 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850684 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850763 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850745 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.850825 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.850777 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.851025 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.851004 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.851217 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.851198 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.851390 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.851366 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.852814 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.852788 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.853086 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.853066 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.853280 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.853257 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.907929 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.907896 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.908222 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.908199 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:22.973181 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:22.973105 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:23.052522 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.052494 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:23.052683 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.052549 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:23.052742 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:23.052685 2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 20:29:23.052742 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:23.052734 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.052721458 +0000 UTC m=+47.530252924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : configmap "kube-rbac-proxy" not found Nov 03 20:29:23.053010 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:23.052974 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:23.053118 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:23.053046 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:27.053027825 +0000 UTC m=+47.530559304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:23.407086 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.407060 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:23.424072 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.424043 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:23.455980 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.455947 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.455993 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456035 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456071 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456121 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456169 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456152 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456412 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456198 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates\") pod \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\" (UID: \"c85654a0-283c-4b2b-bc7c-253a053d9f4d\") " Nov 03 20:29:23.456412 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456373 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.456504 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456415 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:23.456555 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456533 2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-trusted-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.456601 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456553 2572 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c85654a0-283c-4b2b-bc7c-253a053d9f4d-ca-trust-extracted\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.456759 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.456734 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:23.458491 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.458464 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "kube-api-access-b764x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:23.458583 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.458496 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:23.458779 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.458756 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:23.458837 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.458801 2572 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token" pod "c85654a0-283c-4b2b-bc7c-253a053d9f4d" (UID: "c85654a0-283c-4b2b-bc7c-253a053d9f4d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:23.471894 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.471870 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5c54f4d6-pwkct"] Nov 03 20:29:23.514320 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:23.514287 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc63a7c_b9ed_4269_90aa_49a71a94d932.slice/crio-7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b WatchSource:0}: Error finding container 7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b: Status 404 returned error can't find the container with id 7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b Nov 03 20:29:23.557280 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.557251 2572 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-installation-pull-secrets\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.557393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.557287 2572 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-certificates\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.557393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.557305 2572 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-bound-sa-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.557393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.557318 2572 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c85654a0-283c-4b2b-bc7c-253a053d9f4d-image-registry-private-configuration\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.557393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:23.557332 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b764x\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-kube-api-access-b764x\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:24.415389 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.415338 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" event={"ID":"abc63a7c-b9ed-4269-90aa-49a71a94d932","Type":"ContainerStarted","Data":"c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af"} Nov 03 20:29:24.415895 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.415399 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" event={"ID":"abc63a7c-b9ed-4269-90aa-49a71a94d932","Type":"ContainerStarted","Data":"7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b"} Nov 03 20:29:24.415895 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.415424 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:24.415895 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.415547 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-75cb4f9586-ps4d8" Nov 03 20:29:24.533982 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.533930 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" podStartSLOduration=2.533918724 podStartE2EDuration="2.533918724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:24.533197725 +0000 UTC m=+45.010729223" watchObservedRunningTime="2026-11-03 20:29:24.533918724 +0000 UTC m=+45.011450208" Nov 03 20:29:24.701275 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.701204 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-75cb4f9586-ps4d8"] Nov 03 20:29:24.705752 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.705730 2572 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-75cb4f9586-ps4d8"] Nov 03 20:29:24.769060 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:24.769032 2572 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c85654a0-283c-4b2b-bc7c-253a053d9f4d-registry-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:26.098258 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:26.098183 2572 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c85654a0-283c-4b2b-bc7c-253a053d9f4d" path="/var/lib/kubelet/pods/c85654a0-283c-4b2b-bc7c-253a053d9f4d/volumes" Nov 03 20:29:26.111773 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:26.111748 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwd62_f34740ae-e26e-4188-a405-4da96b57553e/dns-node-resolver/0.log" Nov 03 20:29:26.424259 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:26.424227 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" event={"ID":"b91b91f6-e725-4d7b-b472-9e76650979ab","Type":"ContainerStarted","Data":"a40d74ce84976961cdf9b52f3b918c20fd46a14fd894ccde0cedd6f3ddb340e2"} Nov 03 20:29:26.533176 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:26.533128 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" podStartSLOduration=1.9761152119999998 podStartE2EDuration="5.533115513s" totalImagesPullingTime="3.557000301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:26.53172635 +0000 UTC m=+47.009257838" watchObservedRunningTime="2026-11-03 20:29:26.533115513 +0000 UTC m=+47.010646999" Nov 03 20:29:26.837042 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:26.836966 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b5c54f4d6-pwkct_abc63a7c-b9ed-4269-90aa-49a71a94d932/registry/0.log" Nov 03 20:29:27.036790 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.036767 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2j4jr_2ab15921-e819-4567-af4a-9d3684c054ae/node-ca/0.log" Nov 03 20:29:27.088381 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.088303 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:27.088381 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.088349 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:27.088599 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:27.088450 2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 20:29:27.088599 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:27.088507 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls podName:52568b14-d428-42c1-af42-2e15763fc1b3 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:35.088493622 +0000 UTC m=+55.566025088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dpsrk" (UID: "52568b14-d428-42c1-af42-2e15763fc1b3") : secret "insights-runtime-extractor-tls" not found Nov 03 20:29:27.088935 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.088912 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:27.391393 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.391360 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:27.391804 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:27.391520 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle podName:f5ca1597-5b32-4231-97cd-e041e4ff6cd4 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:43.391503523 +0000 UTC m=+63.869035010 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle") pod "router-default-76f9745bc9-fxgzq" (UID: "f5ca1597-5b32-4231-97cd-e041e4ff6cd4") : configmap references non-existent config key: service-ca.crt Nov 03 20:29:27.492022 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.491994 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:27.492178 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.492049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:27.492178 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.492089 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:27.492178 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.492139 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:27.492304 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:27.492222 2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:27.492391 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.492292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:27.492391 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.492342 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:27.492475 ip-10-0-138-52 kubenswrapper[2572]: E1103 20:29:27.492388 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls podName:6b4895c8-b425-4009-a06b-47f1d51b8784 nodeName:}" failed. No retries permitted until 2026-11-03 20:29:43.492371373 +0000 UTC m=+63.969902844 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-627kh" (UID: "6b4895c8-b425-4009-a06b-47f1d51b8784") : secret "cluster-monitoring-operator-tls" not found Nov 03 20:29:27.494606 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.494580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:27.494606 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.494590 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:27.494606 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.494599 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:27.494919 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.494903 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:27.495213 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.495198 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:27.510470 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.510453 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:27.778210 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.778147 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:27.780838 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.780819 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:27.792572 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.792550 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:27.840654 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.840584 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bw59n_3bbdae4d-8429-4f49-8d31-fe00df3d0701/serve-healthcheck-canary/0.log" Nov 03 20:29:27.878977 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:27.878955 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sd2tb"] Nov 03 20:29:27.892204 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:27.892160 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44f243f_ff0c_4e9a_9a91_b0079c2667b6.slice/crio-01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac WatchSource:0}: Error finding container 01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac: Status 404 returned error can't find the container with id 01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac Nov 03 20:29:28.038220 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.038098 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-977vh_b91b91f6-e725-4d7b-b472-9e76650979ab/migrator/0.log" Nov 03 20:29:28.152702 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.152672 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-2b69b"] Nov 03 20:29:28.158897 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:28.158870 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bc681f0_27c6_4cad_8bf6_82835e009976.slice/crio-a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e WatchSource:0}: Error finding container a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e: Status 404 returned error can't find the container with id a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e Nov 03 20:29:28.183599 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.183574 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b"] Nov 03 20:29:28.209676 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.209654 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-595485878c-mqdnk"] Nov 03 20:29:28.215027 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:28.215005 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4382cc48_e16d_4e80_85af_ff5f5d715c7a.slice/crio-c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914 WatchSource:0}: Error finding container c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914: Status 404 returned error can't find the container with id c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914 Nov 03 20:29:28.433935 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.433895 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" event={"ID":"aac3b6b3-61bd-4ee2-b116-61fb0a786896","Type":"ContainerStarted","Data":"1bfa12a04dbcdc5ce6e7748b5660b6fb62d15971dda6c48a9dda8e75384877b7"} Nov 03 20:29:28.435602 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.435572 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595485878c-mqdnk" event={"ID":"4382cc48-e16d-4e80-85af-ff5f5d715c7a","Type":"ContainerStarted","Data":"e050b87363d833b7b8b478adfc65719fa3127c8196a92b4b5e9e8a6da93ec420"} Nov 03 20:29:28.435724 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.435614 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595485878c-mqdnk" event={"ID":"4382cc48-e16d-4e80-85af-ff5f5d715c7a","Type":"ContainerStarted","Data":"c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914"} Nov 03 20:29:28.435724 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.435699 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:28.437052 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.437028 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sd2tb" event={"ID":"e44f243f-ff0c-4e9a-9a91-b0079c2667b6","Type":"ContainerStarted","Data":"01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac"} Nov 03 20:29:28.438525 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.438485 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" event={"ID":"5bc681f0-27c6-4cad-8bf6-82835e009976","Type":"ContainerStarted","Data":"a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e"} Nov 03 20:29:28.439582 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.439558 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-9nws7_5e448fa5-814c-427f-854e-02708388f82a/kube-storage-version-migrator-operator/0.log" Nov 03 20:29:28.440146 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.440120 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" event={"ID":"ff18aebf-b9af-4296-aaad-5d24fd9304ea","Type":"ContainerStarted","Data":"3ec361d3068118415c2835fababcb7a7a92c5c34f0e544c174f3f75da24bbb6a"} Nov 03 20:29:28.552557 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.552506 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-595485878c-mqdnk" podStartSLOduration=38.552493581 podStartE2EDuration="38.552493581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:28.551562819 +0000 UTC m=+49.029094306" watchObservedRunningTime="2026-11-03 20:29:28.552493581 +0000 UTC m=+49.030025067" Nov 03 20:29:28.660547 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:28.660493 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" podStartSLOduration=2.274449841 podStartE2EDuration="7.660479984s" totalImagesPullingTime="5.386030143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:28.659436638 +0000 UTC m=+49.136968124" watchObservedRunningTime="2026-11-03 20:29:28.660479984 +0000 UTC m=+49.138011470" Nov 03 20:29:29.093821 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:29.093789 2572 scope.go:122] "RemoveContainer" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" Nov 03 20:29:29.445827 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:29.445801 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:29:29.446215 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:29.445916 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" event={"ID":"c7a59cf1-454d-4019-9e05-0108e38b8725","Type":"ContainerStarted","Data":"674ca085acd74fd046c80e7b1b1039570261bd594e7dc37c27b77a2a5789d4a1"} Nov 03 20:29:29.446438 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:29.446413 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:29.562732 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:29.562682 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" podStartSLOduration=380.846822059 podStartE2EDuration="6m25.562668373s" totalImagesPullingTime="4.715846314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:29.561697629 +0000 UTC m=+50.039229115" watchObservedRunningTime="2026-11-03 20:29:29.562668373 +0000 UTC m=+50.040199859" Nov 03 20:29:29.958555 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:29.958523 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:31.455342 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:31.455299 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" event={"ID":"5bc681f0-27c6-4cad-8bf6-82835e009976","Type":"ContainerStarted","Data":"66d44e4dabe3e9b73359c9ade9cfbdd4fd8295528ebaf03961272ef5559ae77a"} Nov 03 20:29:31.601420 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:31.601356 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" podStartSLOduration=354.681451352 podStartE2EDuration="5m57.60134226s" totalImagesPullingTime="2.919890908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:31.60078637 +0000 UTC m=+52.078317859" watchObservedRunningTime="2026-11-03 20:29:31.60134226 +0000 UTC m=+52.078873748" Nov 03 20:29:32.459798 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:32.459761 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" event={"ID":"aac3b6b3-61bd-4ee2-b116-61fb0a786896","Type":"ContainerStarted","Data":"59f14a88ff6ec8cfc79f598118892b67181b7fda49c90e004ea5c0ec0b3907ca"} Nov 03 20:29:32.460178 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:32.459823 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" event={"ID":"aac3b6b3-61bd-4ee2-b116-61fb0a786896","Type":"ContainerStarted","Data":"8c226fa1de9480f93d34c4a98ee5b1dc354d48108b1619ab251ac72e58615ea4"} Nov 03 20:29:32.570964 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:32.570907 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" podStartSLOduration=380.860329326 podStartE2EDuration="6m24.570890115s" totalImagesPullingTime="3.710560789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:32.570202989 +0000 UTC m=+53.047734477" watchObservedRunningTime="2026-11-03 20:29:32.570890115 +0000 UTC m=+53.048421621" Nov 03 20:29:33.464678 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:33.464645 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sd2tb" event={"ID":"e44f243f-ff0c-4e9a-9a91-b0079c2667b6","Type":"ContainerStarted","Data":"4c2526d93c957041497dea71993d24635779591844168c133ba7ca39bfed3502"} Nov 03 20:29:33.465060 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:33.464791 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:33.465060 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:33.464832 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sd2tb" event={"ID":"e44f243f-ff0c-4e9a-9a91-b0079c2667b6","Type":"ContainerStarted","Data":"b6ef44f929756adc2e8a5a900c82f007ad4455d3c5753cd17a72869b7f562364"} Nov 03 20:29:33.575497 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:33.575447 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-sd2tb" podStartSLOduration=17.478198454 podStartE2EDuration="22.57543419s" totalImagesPullingTime="5.097235736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:33.575138379 +0000 UTC m=+54.052669868" watchObservedRunningTime="2026-11-03 20:29:33.57543419 +0000 UTC m=+54.052965677" Nov 03 20:29:35.160574 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:35.160528 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:35.162979 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:35.162956 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:35.271050 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:35.271016 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:35.295639 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:35.295598 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:35.678228 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:35.678193 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dpsrk"] Nov 03 20:29:35.684665 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:35.684616 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52568b14_d428_42c1_af42_2e15763fc1b3.slice/crio-b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a WatchSource:0}: Error finding container b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a: Status 404 returned error can't find the container with id b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a Nov 03 20:29:36.477868 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:36.477832 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dpsrk" event={"ID":"52568b14-d428-42c1-af42-2e15763fc1b3","Type":"ContainerStarted","Data":"82de71a00fb292a46b8795fbc30e6cba876a1ec5e41f269ad024feddd91059b7"} Nov 03 20:29:36.478256 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:36.477888 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dpsrk" event={"ID":"52568b14-d428-42c1-af42-2e15763fc1b3","Type":"ContainerStarted","Data":"b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a"} Nov 03 20:29:37.482589 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:37.482555 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dpsrk" event={"ID":"52568b14-d428-42c1-af42-2e15763fc1b3","Type":"ContainerStarted","Data":"ee3179e778acf370fe557179c2de7377b04041b862a9a36c36323956ca974eb1"} Nov 03 20:29:39.491676 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:39.491640 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dpsrk" event={"ID":"52568b14-d428-42c1-af42-2e15763fc1b3","Type":"ContainerStarted","Data":"7056dc9d8e008a385e71b9ed5aee5fd8256b485187cf5a152d2b9084b66e5853"} Nov 03 20:29:39.654731 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:39.654681 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dpsrk" podStartSLOduration=17.425973578 podStartE2EDuration="20.654666165s" totalImagesPullingTime="3.228692587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 20:29:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:39.60757412 +0000 UTC m=+60.085105607" watchObservedRunningTime="2026-11-03 20:29:39.654666165 +0000 UTC m=+60.132197651" Nov 03 20:29:42.921586 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:42.921544 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-595485878c-mqdnk"] Nov 03 20:29:42.977770 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:42.977740 2572 patch_prober.go:29] interesting pod/image-registry-7b5c54f4d6-pwkct container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 20:29:42.977912 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:42.977798 2572 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" podUID="abc63a7c-b9ed-4269-90aa-49a71a94d932" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:43.429560 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.429519 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:43.430196 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.430174 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:43.471513 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.471489 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:43.530476 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.530450 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:43.532905 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.532883 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:43.689835 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.689762 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v9mbf\"" Nov 03 20:29:43.691955 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.691937 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:43.717216 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.717193 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mfn79\"" Nov 03 20:29:43.719281 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:43.719259 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:44.114650 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.114552 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-627kh"] Nov 03 20:29:44.118390 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.118361 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76f9745bc9-fxgzq"] Nov 03 20:29:44.121149 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:44.121123 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4895c8_b425_4009_a06b_47f1d51b8784.slice/crio-abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6 WatchSource:0}: Error finding container abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6: Status 404 returned error can't find the container with id abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6 Nov 03 20:29:44.171098 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:44.171069 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ca1597_5b32_4231_97cd_e041e4ff6cd4.slice/crio-052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47 WatchSource:0}: Error finding container 052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47: Status 404 returned error can't find the container with id 052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47 Nov 03 20:29:44.511107 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.511071 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" event={"ID":"6b4895c8-b425-4009-a06b-47f1d51b8784","Type":"ContainerStarted","Data":"abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6"} Nov 03 20:29:44.512512 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.512486 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" event={"ID":"f5ca1597-5b32-4231-97cd-e041e4ff6cd4","Type":"ContainerStarted","Data":"41b444f9b51a78ac1f64402f89a625f8781b10eb727d733af02194c6715c8fb5"} Nov 03 20:29:44.512652 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.512526 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" event={"ID":"f5ca1597-5b32-4231-97cd-e041e4ff6cd4","Type":"ContainerStarted","Data":"052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47"} Nov 03 20:29:44.627538 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.627482 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" podStartSLOduration=392.627468142 podStartE2EDuration="6m32.627468142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:44.627116234 +0000 UTC m=+65.104647723" watchObservedRunningTime="2026-11-03 20:29:44.627468142 +0000 UTC m=+65.104999628" Nov 03 20:29:44.720081 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.720044 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:44.722688 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.722665 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:44.943783 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.943754 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:44.943950 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.943803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:44.992345 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.992314 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:29:44.992345 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.992326 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:29:44.996615 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.996581 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:44.996734 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:44.996679 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:45.294842 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.294771 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\"" Nov 03 20:29:45.296898 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.296867 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:45.301778 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.301760 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\"" Nov 03 20:29:45.304848 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.304831 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:45.425818 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.425787 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:45.515904 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.515869 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:45.517061 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.517036 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:45.680270 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.680238 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5khcq"] Nov 03 20:29:45.730006 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:45.729975 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfs2r"] Nov 03 20:29:45.736773 ip-10-0-138-52 kubenswrapper[2572]: W1103 20:29:45.736746 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea25f895_6d8c_4bf1_abd4_6e29c4a772e7.slice/crio-f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b WatchSource:0}: Error finding container f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b: Status 404 returned error can't find the container with id f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b Nov 03 20:29:46.521727 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:46.521687 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5khcq" event={"ID":"c947407f-1cc6-4c01-997e-99e19a3c139c","Type":"ContainerStarted","Data":"dcfffc236ff4c616c2373442f45d57a5b0f5e26b840df411bf49d832eeb3a944"} Nov 03 20:29:46.522894 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:46.522869 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfs2r" event={"ID":"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7","Type":"ContainerStarted","Data":"f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b"} Nov 03 20:29:47.529368 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:47.529329 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" event={"ID":"6b4895c8-b425-4009-a06b-47f1d51b8784","Type":"ContainerStarted","Data":"d3af7e7fdcc769e51b3552953506adbb29149d2f68cea5cd6320236c8a988cb7"} Nov 03 20:29:47.639634 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:47.639575 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" podStartSLOduration=398.92567143 podStartE2EDuration="6m41.63956117s" totalImagesPullingTime="2.71388974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:47.638977566 +0000 UTC m=+68.116509054" watchObservedRunningTime="2026-11-03 20:29:47.63956117 +0000 UTC m=+68.117092657" Nov 03 20:29:48.536708 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:48.536659 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfs2r" event={"ID":"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7","Type":"ContainerStarted","Data":"b451420d00a0fabbf481b9bd602e5f7f7f63fa71f4cde6b325d9773505d11f61"} Nov 03 20:29:48.537140 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:48.536730 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfs2r" event={"ID":"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7","Type":"ContainerStarted","Data":"c3a2a44efe5933d85f6ef159187844314099bca5aad2bdb7933ffb76c0a2fa95"} Nov 03 20:29:48.648875 ip-10-0-138-52 kubenswrapper[2572]: I1103 20:29:48.648812 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vfs2r" podStartSLOduration=66.569179345 podStartE2EDuration="1m8.64879591s" totalImagesPullingTime="2.079616565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:29:48.647298377 +0000 UTC m=+69.124829864" watchObservedRunningTime="2026-11-03 20:29:48.64879591 +0000 UTC m=+69.126327423" Nov 03 20:29:51.864917 ip-10-0-138-52 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 20:29:51.879010 ip-10-0-138-52 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 20:29:51.879180 ip-10-0-138-52 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 20:29:51.879228 ip-10-0-138-52 systemd[1]: kubelet.service: Consumed 4.587s CPU time. Nov 03 20:29:51.889276 ip-10-0-138-52 systemd[1]: Starting Kubernetes Kubelet... Nov 03 20:29:51.964350 ip-10-0-138-52 kubenswrapper[5932]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:29:51.964350 ip-10-0-138-52 kubenswrapper[5932]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 20:29:51.964350 ip-10-0-138-52 kubenswrapper[5932]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:29:51.964350 ip-10-0-138-52 kubenswrapper[5932]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 20:29:51.966711 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966613 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.966711 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966709 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.966711 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966713 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966717 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966720 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966723 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966726 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966736 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966739 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966742 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966745 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966747 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966750 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966756 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966759 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966761 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966764 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966768 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966773 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966776 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966779 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.966817 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966782 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966785 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966787 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966790 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966793 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966795 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966798 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966801 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966803 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966806 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966809 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966811 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966814 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966822 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966826 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966828 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966831 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966834 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966837 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.967299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966839 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966842 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966845 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966847 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966850 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966852 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966855 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966857 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966859 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966862 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966865 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966868 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966870 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966873 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966876 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966879 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966881 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966884 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966887 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966889 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.967856 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966892 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966895 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966898 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966902 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966906 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966909 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966912 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966920 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966923 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966925 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966928 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966931 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966933 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966936 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966938 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966941 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966944 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966946 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966949 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966951 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.968370 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966954 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966956 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966959 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966961 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966964 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966966 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966969 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966972 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966975 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966978 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966980 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966983 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966986 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966988 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966992 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966994 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966997 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.966999 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967002 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.968894 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967004 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967014 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967018 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967021 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967023 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967026 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967028 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967879 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967888 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967891 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967894 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967897 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967900 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967903 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967906 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967908 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967911 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967913 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967916 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.969375 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967922 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967926 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967930 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967933 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967936 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967938 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967941 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967944 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967947 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967950 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967953 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967955 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967958 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967961 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967963 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967972 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967975 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967979 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967981 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.969902 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967984 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967987 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967989 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967992 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967994 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967997 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.967999 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968002 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968005 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968008 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968010 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968013 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968015 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968018 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968021 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968023 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968026 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968029 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968031 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968034 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.970595 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968037 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968040 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968042 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968045 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968047 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968050 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968053 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968055 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968058 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968067 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968070 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968073 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968076 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968078 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968081 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968084 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968086 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968089 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968091 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.971256 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968094 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968096 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968099 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968101 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968104 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968106 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968109 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968111 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968114 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968116 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968119 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968121 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968124 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968126 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968129 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968132 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968135 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968137 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968140 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.971778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968142 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968145 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968147 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968150 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968158 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968161 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968164 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968166 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968169 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968188 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968192 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968195 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968198 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968201 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968204 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968207 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968210 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968799 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968810 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.972269 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968814 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968817 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968820 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968823 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968825 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968829 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968831 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968834 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968837 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968839 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968842 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968844 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968847 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968850 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968852 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968855 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968858 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968866 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968870 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968872 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.972769 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968875 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968878 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968881 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968883 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968886 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968889 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968892 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968894 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968897 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968900 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968903 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968909 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968912 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968915 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968917 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968920 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968922 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968925 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968927 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968930 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.973294 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968933 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968935 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968938 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968940 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968943 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968945 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968948 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968950 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968953 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968956 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968964 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968967 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968969 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968972 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968975 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968977 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968980 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968982 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968984 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.973887 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968987 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968990 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968992 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968995 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.968999 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969001 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969004 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969007 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969009 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969012 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969015 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969018 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969021 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969023 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969026 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969029 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969031 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969034 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969037 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.974362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969040 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969042 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969045 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969047 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969050 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969058 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969061 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969063 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969066 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969068 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969071 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969073 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969076 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969079 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969081 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969084 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969086 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969090 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969093 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969096 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.974861 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969098 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969101 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969107 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969110 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969113 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.969116 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969213 5932 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969223 5932 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969240 5932 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969248 5932 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969253 5932 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969256 5932 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969260 5932 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969264 5932 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969268 5932 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969271 5932 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969274 5932 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969278 5932 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969289 5932 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969293 5932 flags.go:64] FLAG: --cgroup-root="" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969296 5932 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969299 5932 flags.go:64] FLAG: --client-ca-file="" Nov 03 20:29:51.975349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969302 5932 flags.go:64] FLAG: --cloud-provider="external" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969305 5932 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969312 5932 flags.go:64] FLAG: --cluster-domain="" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969315 5932 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969318 5932 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969321 5932 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969324 5932 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969333 5932 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969336 5932 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969340 5932 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969343 5932 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969346 5932 flags.go:64] FLAG: --contention-profiling="false" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969348 5932 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969352 5932 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969355 5932 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969358 5932 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969362 5932 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969365 5932 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969368 5932 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969377 5932 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969381 5932 flags.go:64] FLAG: --enable-server="true" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969384 5932 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969391 5932 flags.go:64] FLAG: --event-burst="100" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969394 5932 flags.go:64] FLAG: --event-qps="50" Nov 03 20:29:51.975939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969397 5932 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969400 5932 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969403 5932 flags.go:64] FLAG: --eviction-hard="" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969407 5932 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969410 5932 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969419 5932 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969422 5932 flags.go:64] FLAG: --eviction-soft="" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969425 5932 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969428 5932 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969431 5932 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969434 5932 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969437 5932 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969440 5932 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969443 5932 flags.go:64] FLAG: --feature-gates="" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969447 5932 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969451 5932 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969454 5932 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969458 5932 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969461 5932 flags.go:64] FLAG: --healthz-port="10248" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969464 5932 flags.go:64] FLAG: --help="false" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969467 5932 flags.go:64] FLAG: --hostname-override="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969470 5932 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969473 5932 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 20:29:51.976535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969476 5932 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969479 5932 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969482 5932 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969485 5932 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969488 5932 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969491 5932 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969494 5932 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969497 5932 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969500 5932 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969503 5932 flags.go:64] FLAG: --kube-reserved="" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969506 5932 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969508 5932 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969511 5932 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969514 5932 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969517 5932 flags.go:64] FLAG: --lock-file="" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969526 5932 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969529 5932 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969532 5932 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969540 5932 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969543 5932 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969546 5932 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969549 5932 flags.go:64] FLAG: --logging-format="text" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969552 5932 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969555 5932 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 20:29:51.977122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969559 5932 flags.go:64] FLAG: --manifest-url="" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969562 5932 flags.go:64] FLAG: --manifest-url-header="" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969567 5932 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969570 5932 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969573 5932 flags.go:64] FLAG: --max-pods="110" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969577 5932 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969580 5932 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969583 5932 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969586 5932 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969589 5932 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969592 5932 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969595 5932 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969601 5932 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969605 5932 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969608 5932 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969611 5932 flags.go:64] FLAG: --pod-cidr="" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969614 5932 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969617 5932 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969633 5932 flags.go:64] FLAG: --pods-per-core="0" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969636 5932 flags.go:64] FLAG: --port="10250" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969640 5932 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969643 5932 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f38ed42849de5b04" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969646 5932 flags.go:64] FLAG: --qos-reserved="" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969649 5932 flags.go:64] FLAG: --read-only-port="10255" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969664 5932 flags.go:64] FLAG: --register-node="true" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969668 5932 flags.go:64] FLAG: --register-with-taints="" Nov 03 20:29:51.977762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969678 5932 flags.go:64] FLAG: --registry-burst="10" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969682 5932 flags.go:64] FLAG: --registry-qps="5" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969685 5932 flags.go:64] FLAG: --reserved-cpus="" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969688 5932 flags.go:64] FLAG: --reserved-memory="" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969691 5932 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969694 5932 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969699 5932 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969703 5932 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969706 5932 flags.go:64] FLAG: --runonce="false" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969709 5932 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969712 5932 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969715 5932 flags.go:64] FLAG: --seccomp-default="false" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969718 5932 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969721 5932 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969724 5932 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969727 5932 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969730 5932 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969733 5932 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969736 5932 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969739 5932 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969742 5932 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969745 5932 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969748 5932 flags.go:64] FLAG: --system-cgroups="" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969751 5932 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969756 5932 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 20:29:51.978470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969759 5932 flags.go:64] FLAG: --tls-cert-file="" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969762 5932 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969769 5932 flags.go:64] FLAG: --tls-min-version="" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969772 5932 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969775 5932 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969778 5932 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969787 5932 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969792 5932 flags.go:64] FLAG: --v="2" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969796 5932 flags.go:64] FLAG: --version="false" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969800 5932 flags.go:64] FLAG: --vmodule="" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969804 5932 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.969807 5932 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970323 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970347 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970352 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970357 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970369 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970373 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970377 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970382 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970386 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970391 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970395 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.979332 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970399 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970403 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970408 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970412 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970416 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970425 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970430 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970434 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970438 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970442 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970446 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970450 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970455 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970459 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970465 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970472 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970480 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970484 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970493 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970497 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.979937 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970502 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970506 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970512 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970517 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970522 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970526 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970530 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970535 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970539 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970543 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970557 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970566 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970570 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970574 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970578 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970583 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970587 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970592 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970596 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.980723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970601 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970606 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970610 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970614 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970639 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970644 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970648 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970653 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970657 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970667 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970672 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970676 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970680 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970685 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970689 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970841 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970847 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970858 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970864 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.981266 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970871 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970893 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970899 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970903 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970908 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970912 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970916 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970920 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970924 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970929 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970939 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970943 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970948 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970952 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970957 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970964 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970967 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970970 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.970973 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971009 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.981803 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971013 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971028 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971033 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971133 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971137 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971140 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971143 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971145 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971148 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971151 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971153 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971156 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971159 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971161 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971164 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971167 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.982305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971169 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.971203 5932 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971266 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971274 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971278 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971281 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971283 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971287 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971290 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971293 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971297 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971299 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971303 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971308 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971311 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971316 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.982785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971318 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971321 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971328 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971333 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971335 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971338 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971341 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971344 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971346 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971349 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971352 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971354 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971357 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971359 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971362 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971365 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971367 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971370 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971372 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971375 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.983262 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971378 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971380 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971383 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971385 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971388 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971391 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971393 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971396 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971398 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971401 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971403 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971406 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971408 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971411 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971415 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971419 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971422 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971425 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971427 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.984006 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971430 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971433 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971435 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971438 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971441 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971443 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971446 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971449 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971451 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971454 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971456 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971459 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971461 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971464 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971466 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971469 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971472 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971474 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971477 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971479 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.984489 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971482 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971485 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971496 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971499 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971502 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971505 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971509 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971515 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971519 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971522 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971525 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971527 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971530 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971532 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971535 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971538 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971540 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971543 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971546 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.985074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971548 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971550 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971553 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971556 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971558 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971561 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971563 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971566 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971568 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971571 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971575 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971578 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971580 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.971583 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.975431 5932 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.975448 5932 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.975469 5932 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.975474 5932 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975526 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975530 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.985729 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975534 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975536 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975539 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975542 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975545 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975548 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975551 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975554 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975556 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975559 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975562 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975565 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975568 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975571 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975574 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975576 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975579 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975582 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975584 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975587 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.986559 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975589 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975593 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975595 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975599 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975602 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975604 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975607 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975610 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975613 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975616 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975631 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975636 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975650 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975653 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975656 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975658 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975661 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975664 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975667 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975670 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.987235 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975672 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975675 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975677 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975680 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975683 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975685 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975688 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975690 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975693 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975696 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975698 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975701 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975704 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975706 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975709 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975712 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975715 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975717 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975720 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975723 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.987822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975726 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975729 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975732 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975734 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975737 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975739 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975742 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975745 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975747 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975750 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975753 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975755 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975758 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975761 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975763 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975766 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975768 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975772 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975776 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.988316 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975780 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975782 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975785 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975787 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975792 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975795 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975798 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975801 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975804 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975807 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975810 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975813 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975815 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975818 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975821 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975824 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975827 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975829 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975832 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.988802 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975834 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975837 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975839 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975842 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975844 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.975847 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.975881 5932 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976016 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976020 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976024 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976027 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976030 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976033 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976036 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976039 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976042 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.989337 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976044 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976047 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976049 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976052 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976055 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976057 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976060 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976062 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976065 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976067 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976070 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976073 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976076 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976078 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976081 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976084 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976086 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976089 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976091 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.989758 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976094 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976096 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976099 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976101 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976104 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976106 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976109 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976111 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976114 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976116 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976119 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976122 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976124 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976127 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976129 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976132 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976134 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976138 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976142 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.990230 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976145 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976147 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976150 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976153 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976159 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976162 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976166 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976171 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976174 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976177 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976180 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976182 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976185 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976187 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976190 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976192 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976195 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976197 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976200 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976203 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.990755 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976205 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976208 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976211 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976214 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976216 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976219 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976222 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976225 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976227 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976230 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976232 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976235 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976238 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976240 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976243 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976246 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976250 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976252 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976255 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976258 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.991244 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976261 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976264 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976266 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976269 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976271 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976274 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976277 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976280 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976282 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976285 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976288 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976290 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976293 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976295 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976298 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976301 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976303 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976306 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.991934 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976309 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.976333 5932 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976366 5932 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976370 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976373 5932 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976376 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976379 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976381 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976384 5932 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976387 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976390 5932 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976393 5932 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976396 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976399 5932 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976401 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976404 5932 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 20:29:51.992382 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976407 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976410 5932 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976412 5932 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976415 5932 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976418 5932 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976420 5932 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976423 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976425 5932 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976428 5932 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976430 5932 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976433 5932 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976436 5932 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976438 5932 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976441 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976443 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976446 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976448 5932 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976451 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976454 5932 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976456 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 20:29:51.992793 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976459 5932 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976461 5932 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976464 5932 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976466 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976469 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976472 5932 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976475 5932 feature_gate.go:386] unrecognized feature gate: Example Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976478 5932 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976481 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976484 5932 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976486 5932 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976489 5932 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976492 5932 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976495 5932 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976498 5932 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976501 5932 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976503 5932 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976506 5932 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976508 5932 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976511 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 20:29:51.993323 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976513 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976516 5932 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976518 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976521 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976524 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976526 5932 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976529 5932 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976531 5932 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976534 5932 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976537 5932 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976539 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976542 5932 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976545 5932 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976548 5932 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976551 5932 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976553 5932 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976556 5932 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976559 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976561 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 20:29:51.993988 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976565 5932 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976568 5932 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976570 5932 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976574 5932 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976578 5932 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976581 5932 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976584 5932 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976587 5932 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976590 5932 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976592 5932 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976595 5932 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976599 5932 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976604 5932 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976607 5932 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976610 5932 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976612 5932 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976617 5932 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976633 5932 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976638 5932 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 20:29:51.994824 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976641 5932 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976644 5932 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976647 5932 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976649 5932 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976652 5932 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976655 5932 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976657 5932 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976660 5932 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976663 5932 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976665 5932 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976667 5932 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976670 5932 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976673 5932 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:29:51.976677 5932 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.976736 5932 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.977666 5932 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.977721 5932 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.977904 5932 server.go:1014] "Starting client certificate rotation" Nov 03 20:29:51.995299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.978046 5932 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.978098 5932 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 20:23:40 +0000 UTC" deadline="2028-05-12 17:28:36.849238412 +0000 UTC" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.978133 5932 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13340h58m44.871107204s" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.978370 5932 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.979392 5932 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.980923 5932 log.go:25] "Validated CRI v1 runtime API" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.982772 5932 log.go:25] "Validated CRI v1 image API" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.983464 5932 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 20:29:51.995769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.987311 5932 fs.go:135] Filesystem UUIDs: map[23a14c2e-5782-43ad-b0dd-97fcc04a8e92:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 94badfe8-46b9-4452-84f2-53d432bfcfc0:/dev/nvme0n1p3] Nov 03 20:29:51.996017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.987329 5932 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac/userdata/shm major:0 minor:389 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47/userdata/shm major:0 minor:478 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/093b3fa970ad305da00cbb434365c1b3bfbe2da80a8faaf908a1434b2f88a1f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/093b3fa970ad305da00cbb434365c1b3bfbe2da80a8faaf908a1434b2f88a1f5/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bfa12a04dbcdc5ce6e7748b5660b6fb62d15971dda6c48a9dda8e75384877b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bfa12a04dbcdc5ce6e7748b5660b6fb62d15971dda6c48a9dda8e75384877b7/userdata/shm major:0 minor:393 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6/userdata/shm major:0 minor:476 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a/userdata/shm major:0 minor:455 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914/userdata/shm major:0 minor:401 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcfffc236ff4c616c2373442f45d57a5b0f5e26b840df411bf49d832eeb3a944/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcfffc236ff4c616c2373442f45d57a5b0f5e26b840df411bf49d832eeb3a944/userdata/shm major:0 minor:494 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b/userdata/shm major:0 minor:492 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9/userdata/shm major:0 minor:348 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02bb73c6-e6e2-4d77-8458-20ba459b831a/volumes/kubernetes.io~projected/kube-api-access-qxxd9:{mountpoint:/var/lib/kubelet/pods/02bb73c6-e6e2-4d77-8458-20ba459b831a/volumes/kubernetes.io~projected/kube-api-access-qxxd9 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab15921-e819-4567-af4a-9d3684c054ae/volumes/kubernetes.io~projected/kube-api-access-bwdln:{mountpoint:/var/lib/kubelet/pods/2ab15921-e819-4567-af4a-9d3684c054ae/volumes/kubernetes.io~projected/kube-api-access-bwdln major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3086db43-8d5a-4196-ab9e-8c7f7832c232/volumes/kubernetes.io~projected/kube-api-access-jwmm8:{mountpoint:/var/lib/kubelet/pods/3086db43-8d5a-4196-ab9e-8c7f7832c232/volumes/kubernetes.io~projected/kube-api-access-jwmm8 major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3086db43-8d5a-4196-ab9e-8c7f7832c232/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3086db43-8d5a-4196-ab9e-8c7f7832c232/volumes/kubernetes.io~secret/signing-key major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/311c1fbb-32b5-4d7d-934b-28b68c8d48cb/volumes/kubernetes.io~projected/kube-api-access-9qz8f:{mountpoint:/var/lib/kubelet/pods/311c1fbb-32b5-4d7d-934b-28b68c8d48cb/volumes/kubernetes.io~projected/kube-api-access-9qz8f major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bbdae4d-8429-4f49-8d31-fe00df3d0701/volumes/kubernetes.io~projected/kube-api-access-j5l82:{mountpoint:/var/lib/kubelet/pods/3bbdae4d-8429-4f49-8d31-fe00df3d0701/volumes/kubernetes.io~projected/kube-api-access-j5l82 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3bbdae4d-8429-4f49-8d31-fe00df3d0701/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3bbdae4d-8429-4f49-8d31-fe00df3d0701/volumes/kubernetes.io~secret/cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/kube-api-access-9f59t:{mountpoint:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/kube-api-access-9f59t major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/registry-tls major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a/volumes/kubernetes.io~projected/kube-api-access-ffww8:{mountpoint:/var/lib/kubelet/pods/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a/volumes/kubernetes.io~projected/kube-api-access-ffww8 major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52568b14-d428-42c1-af42-2e15763fc1b3/volumes/kubernetes.io~projected/kube-api-access-smpv4:{mountpoint:/var/lib/kubelet/pods/52568b14-d428-42c1-af42-2e15763fc1b3/volumes/kubernetes.io~projected/kube-api-access-smpv4 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52568b14-d428-42c1-af42-2e15763fc1b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/52568b14-d428-42c1-af42-2e15763fc1b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/536dddb8-cc7a-4b68-a591-7a19d1431afe/volumes/kubernetes.io~projected/kube-api-access-qnfbc:{mountpoint:/var/lib/kubelet/pods/536dddb8-cc7a-4b68-a591-7a19d1431afe/volumes/kubernetes.io~projected/kube-api-access-qnfbc major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bc681f0-27c6-4cad-8bf6-82835e009976/volumes/kubernetes.io~secret/networking-console-plugin-cert:{mountpoint:/var/lib/kubelet/pods/5bc681f0-27c6-4cad-8bf6-82835e009976/volumes/kubernetes.io~secret/networking-console-plugin-cert major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e448fa5-814c-427f-854e-02708388f82a/volumes/kubernetes.io~projected/kube-api-access-j877w:{mountpoint:/var/lib/kubelet/pods/5e448fa5-814c-427f-854e-02708388f82a/volumes/kubernetes.io~projected/kube-api-access-j877w major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e448fa5-814c-427f-854e-02708388f82a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/5e448fa5-814c-427f-854e-02708388f82a/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750/volumes/kubernetes.io~projected/kube-api-access-6htdx:{mountpoint:/var/lib/kubelet/pods/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750/volumes/kubernetes.io~projected/kube-api-access-6htdx major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ab99b87-827c-4aba-a47e-b86a847ca383/volumes/kubernetes.io~projected/kube-api-access-d942d:{mountpoint:/var/lib/kubelet/pods/6ab99b87-827c-4aba-a47e-b86a847ca383/volumes/kubernetes.io~projected/kube-api-access-d942d major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4895c8-b425-4009-a06b-47f1d51b8784/volumes/kubernetes.io~projected/kube-api-access-8lcbr:{mountpoint:/var/lib/kubelet/pods/6b4895c8-b425-4009-a06b-47f1d51b8784/volumes/kubernetes.io~projected/kube-api-access-8lcbr major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4895c8-b425-4009-a06b-47f1d51b8784/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls:{mountpoint:/var/lib/kubelet/pods/6b4895c8-b425-4009-a06b-47f1d51b8784/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls major:0 minor:472 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volumes/kubernetes.io~projected/kube-api-access-pbpnq:{mountpoint:/var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volumes/kubernetes.io~projected/kube-api-access-pbpnq major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3/volumes/kubernetes.io~projected/kube-api-access-zgm99:{mountpoint:/var/lib/kubelet/pods/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3/volumes/kubernetes.io~projected/kube-api-access-zgm99 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98667931-42d6-42d2-b335-57b2d19d7dfc/volumes/kubernetes.io~projected/kube-api-access-p49dx:{mountpoint:/var/lib/kubelet/pods/98667931-42d6-42d2-b335-57b2d19d7dfc/volumes/kubernetes.io~projected/kube-api-access-p49dx major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98667931-42d6-42d2-b335-57b2d19d7dfc/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/98667931-42d6-42d2-b335-57b2d19d7dfc/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7b21223-f0b4-492b-8856-233fa4b1cd58/volumes/kubernetes.io~projected/kube-api-access-9t97p:{mountpoint:/var/lib/kubelet/pods/a7b21223-f0b4-492b-8856-233fa4b1cd58/volumes/kubernetes.io~projected/kube-api-access-9t97p major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a995cca6-542f-406e-a938-aa58715eac9a/volumes/kubernetes.io~projected/kube-api-access-4q4l5:{mountpoint:/var/lib/kubelet/pods/a995cca6-542f-406e-a938-aa58715eac9a/volumes/kubernetes.io~projected/kube-api-access-4q4l5 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aac3b6b3-61bd-4ee2-b116-61fb0a786896/volumes/kubernetes.io~projected/kube-api-access-l82xf:{mountpoint:/var/lib/kubelet/pods/aac3b6b3-61bd-4ee2-b116-61fb0a786896/volumes/kubernetes.io~projected/kube-api-access-l82xf major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aac3b6b3-61bd-4ee2-b116-61fb0a786896/volumes/kubernetes.io~secret/samples-operator-tls:{mountpoint:/var/lib/kubelet/pods/aac3b6b3-61bd-4ee2-b116-61fb0a786896/volumes/kubernetes.io~secret/samples-operator-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/kube-api-access-qdwdc:{mountpoint:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/kube-api-access-qdwdc major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/registry-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~projected/kube-api-access-wzs87:{mountpoint:/var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~projected/kube-api-access-wzs87 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4/volumes/kubernetes.io~projected/kube-api-access-z2rc9:{mountpoint:/var/lib/kubelet/pods/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4/volumes/kubernetes.io~projected/kube-api-access-z2rc9 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b91b91f6-e725-4d7b-b472-9e76650979ab/volumes/kubernetes.io~projected/kube-api-access-kvqhp:{mountpoint:/var/lib/kubelet/pods/b91b91f6-e725-4d7b-b472-9e76650979ab/volumes/kubernetes.io~projected/kube-api-access-kvqhp major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7a59cf1-454d-4019-9e05-0108e38b8725/volumes/kubernetes.io~projected/kube-api-access-b9skt:{mountpoint:/var/lib/kubelet/pods/c7a59cf1-454d-4019-9e05-0108e38b8725/volumes/kubernetes.io~projected/kube-api-access-b9skt major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7a59cf1-454d-4019-9e05-0108e38b8725/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c7a59cf1-454d-4019-9e05-0108e38b8725/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c947407f-1cc6-4c01-997e-99e19a3c139c/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c947407f-1cc6-4c01-997e-99e19a3c139c/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:491 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e44f243f-ff0c-4e9a-9a91-b0079c2667b6/volumes/kubernetes.io~projected/kube-api-access-6pkmm:{mountpoint:/var/lib/kubelet/pods/e44f243f-ff0c-4e9a-9a91-b0079c2667b6/volumes/kubernetes.io~projected/kube-api-access-6pkmm major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e44f243f-ff0c-4e9a-9a91-b0079c2667b6/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e44f243f-ff0c-4e9a-9a91-b0079c2667b6/volumes/kubernetes.io~secret/metrics-tls major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7/volumes/kubernetes.io~projected/kube-api-access-vtlbn:{mountpoint:/var/lib/kubelet/pods/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7/volumes/kubernetes.io~projected/kube-api-access-vtlbn major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7/volumes/kubernetes.io~secret/metrics-certs major:0 minor:486 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea833ecb-ee75-45fd-b689-2b7c0c9a711c/volumes/kubernetes.io~projected/kube-api-access-swnlx:{mountpoint:/var/lib/kubelet/pods/ea833ecb-ee75-45fd-b689-2b7c0c9a711c/volumes/kubernetes.io~projected/kube-api-access-swnlx major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea833ecb-ee75-45fd-b689-2b7c0c9a711c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ea833ecb-ee75-45fd-b689-2b7c0c9a711c/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eee402df-6e0a-4cac-a025-b981a2ee6329/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/eee402df-6e0a-4cac-a025-b981a2ee6329/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f34740ae-e26e-4188-a405-4da96b57553e/volumes/kubernetes.io~projected/kube-api-access-rzg7d:{mountpoint:/var/lib/kubelet/pods/f34740ae-e26e-4188-a405-4da96b57553e/volumes/kubernetes.io~projected/kube-api-access-rzg7d major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~projected/kube-api-access-nfj4h:{mountpoint:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~projected/kube-api-access-nfj4h major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/default-certificate major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/stats-auth major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe95bd0d-a5b5-44db-97ae-d75a6860a71a/volumes/kubernetes.io~projected/kube-api-access-dlz94:{mountpoint:/var/lib/kubelet/pods/fe95bd0d-a5b5-44db-97ae-d75a6860a71a/volumes/kubernetes.io~projected/kube-api-access-dlz94 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff18aebf-b9af-4296-aaad-5d24fd9304ea/volumes/kubernetes.io~projected/kube-api-access-kv629:{mountpoint:/var/lib/kubelet/pods/ff18aebf-b9af-4296-aaad-5d24fd9304ea/volumes/kubernetes.io~projected/kube-api-access-kv629 major:0 minor:358 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/44c3e0909fd9e1607d3253511fc5b9a2b0250ff80420d96cb7e49a0704f99d85/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/173f727bb3e008512ec1a6792336f4415620051f30ad22a15b4cdec659bbf333/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0d419ec95aa5431268121d59bb5118305ac6275cb4225570cb422e8d3e5b7aae/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/a5c0f5d9bdd42487f42750447834b2bc8f022199e5c3c04928f53c92c9082bb0/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/78b474fb0386731af796cb2af9df29e98a4c77f8f76d06ce7ad10d24dcdde84f/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/90e2e337629bdaaf0156e010d5dcdba823a8b5c142d134003a528c65b90cf9db/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-123:{mountpoint:/var/lib/containers/storage/overlay/4229f4cf0b475903d4b90d0386ed4d1a7db8644dd16d166a8c92e4a134f62a03/merged major:0 minor:123 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0fd1951cd14b090b23ab8669e10aa7036343422675e41a0c70cf2e7f0be4b0e7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/80b484d89721a0e36e3a00f67658afa69b913e6121cac328385c27aaf3d242bd/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/fa5ba301b8da93d1d984e2fc422cb34e5059fe4a91cf90ac6ce0b67ed51fbfce/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/8edd4a2525eb15001c0cd17b1b3ac5d12c37ff35302f414b18f6512428de294e/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/0b8ecc696e8c45269c06348c2f1ec1787df36f2dd71286952e0b765dfa5eb52c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/50f099e4c6acf4bc651c326c4a169509642ae2d2d232fde93edbd021e5187c5a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/3eb53a2e6883a8531ef19bc999a8dc339725296f8bb7cc1e61c725535a58da08/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f75bcd34c6eed3853d82ffc543cfa1fc7a0913b4e57af93439c193dec8c2bd9c/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/74e35d42cc751abb5b9e063650f29330ab4f6e1da722b15427fa7df3f700a8f9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/196e59186d92427ad8eed69962d60e1f94fbee79569f4b8aadc65e44c2343746/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5d38b8cef56bd6b67881b60d928b8402b11e616540e1a6ab0644a6ae8bd6d2b2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5c0ee9be207c5aa3844b23832258f81306b6712fe264828f07c424c90087ab07/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5ede99648444c5e1304c4da076e89b6f300c73238a9b810bc702b46c25a8c0ee/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8cdcf9f2c1c26d0c96446e8690c9897098bc999fcc0d0bc74b03c9b10bf159b5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-202:{mountpoint:/var/lib/containers/storage/overlay/6475b1c243a4602d53e603ee8d00183b54b1f7a9522860ce9d9e18af075b0f0f/merged major:0 minor:202 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/f3a1f64e6167497e492ce8b4d313f33728df351aef820a12ae926425e64b9df6/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/1fd92cad4e32020e1fd718b4f8e57642cba1671dacd0b8057f675effe836a14e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c3dbcb7334e94b241eae09d7a05c69acdcc2bc0c4f216cfa559a19938397ae97/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/4b21ab96028140157ff2c1c6c33d2693f9c41e89edaf3ad3203347e638ed128f/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0e095bb6967442f69e608ea8cff117e1c1dfdd2edbedc9788ea7c0cb16801639/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/66e2b6c441bdb74ad37e743d43945a20ce578b3e6fab95f3d45958fc7ff9695c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c3d1dfc4b170d5e139193d6f23466bd626c6f3f52541edb7906192fa3464e8b2/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/afe8f83b4d864b29de6e2af62d0f059dda36aa9e731717ee91859486608bdfdc/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/ac49f20536f8b105fd0f6a9f8546ba2d7db19c246b269ef0bae6f5f37ff2b0b3/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/e9b2e71740a40b1c11556774a78a41b6a20d2678882cb02b02ea426b7911e75b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e33565398af48e1aeb9c6d3c9c5e610030ab8237eda435aed9be4e7d7072fa9a/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/fb3b5e97d41ebcc42c65fd613aad8424697ab0711d7acf32fc132616b85d424f/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b0b8c14407e5bb77eeba1d9238c609be2ac4282ed2dc79bb3cd2d13f3799acad/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/a16743623d48a856a1e9dfea50efa8315d484ab3fea7e0d25d59f886a618244b/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/22dd1caf4b13dbbba81a930c92b125f72d2b70cc4807bbbaed0b5bef04a61d53/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/c1e1271a0629905f904c6eb58d73b2cba0e4e440e10e96088050c6417cb8866f/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/82d2f7004c34df10c159ee75c46b11068f5b49a4f51e6d468de0658df39c34da/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/6cd8f338179eedb6009e37fbe41ff242ead6b617c07e979b5ac0cee1278a267d/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/e914e19817d12786c499fd7eadcc470c1f0492841f06a1fd554f2c1b42611d9e/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/efadf08b77475b0ec8d71a6bd73131d0c4c0ee2389370fcbcc1c81673292f4f1/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/dfccfb899b547856f8a2436d5cb3d4de1af1d5c7a145c97c32ad670922776240/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/5912f93dfeaf23a1a78f612e868a3c4d8c936e1c422964f23041ef3e7a037aff/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/3cc9971e0149d334b255187c3d0893f8aa536cfffd9afcf8f72da0d44cd097b8/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/6fd5567c7e545ab3d62fc1d548bec8bf5e76f0ab0714ba07f2cf5c0d1b2680bb/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/33c5b3f2d6922accd319655802440bed5acc6f560f38499ab31e2aa2b3117648/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/461e7eb814e9b4020aade6cf3cba1430765b9b620f640182b8f1bba272403d29/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/09ed96d76e745173ddd3c894ee07f1466be2ea85b2b032f1c3c91b0c6119b0ed/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/15d3b055af7769cb9f1885939f1aaf6f0c4870d675a1af5a3f189575aaae6e76/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/afc4df1b4184cc4eb76f387d1fcb643f8e67fa7d638af37d0f15a4a856fd26fa/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/8eda32641944013957b5775fb0dc224efd7f2d9b8d4aa8f0677b617f3afbfb8b/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/82c585d70ab51f6e76edf260fdee3507f47684477370f161b94b46d944046f35/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/b664e70ffd3654e02cc563b36d14ac1fe5968d897aa4f29e870b8e508faf9307/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/de6eb5bdd86e3d92552a8dd66362a5de8f7f582f1c202949737fd9f230897687/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/e8ef0a7adb081055403a1dcac72b4e6d2e7497c035aa9bc3e129181d9e996438/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/e7f6aa2359e5cb332358663e9bb4d37a920da8e70aac6d1b7b05b8defd7b1a37/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/57eea14af66a4205aeeaf5f05cb643a9a38188555651ca316e118b7aa740464c/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/8adc3ff68ab8a52ddf8b7a1fd33a480b8f98c99ac812dc56cf2be3bc527a2775/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/f3f97805d468805d0c89460123c85207d00119c8d72e9884bd0a09f0e7c05921/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/9638b57a8c1862b247f7535375717f14876b2fcaf0d5aa7ebe0b668c98a4531b/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/241a1e88dd4411e2d573275cd094f952d8db6cf912153241474adf31f5b9f1c8/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/c81d83c5f6bfaec341d2d9dc5caf827a335e791148dd34b5e8e0334df082a1c8/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/7bda236c6a4cf4a36da955929887cacac1a87a7e19f6dbf63014f73aa8bf3756/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/f6839dd0c990d9e24cda81562a91ee49e6814efb8bb2533581b70b9784b79ce1/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-461:{mountpoint:/var/lib/containers/storage/overlay/117f913c9cf18bb40beee7cd5ba8c3e2f343d86e06e4542738e19b11bdf026e7/merged major:0 minor:461 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/8e1fe9df17d1da85855b3738eb54f02d4867a29185cf5201bb75e3640e1c5306/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/8dba5b892575f0baf521f6158d6af11da7d6cadae9b34027c799fcaa6c9fa04a/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/8760677807d9c58bbea41b98a59633b1d9f62ae644e1212ae413b6a9f771ddaf/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/c99ad22bbe7d00f960d195a333a2a5cb07e0f51031d545c0d821885299d39148/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/377398ff038b0ed063a848304990f38e5d79071c9afecdeef4799149724d19a1/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/fa3ad0a6e5771e0b9e0b2eb127f6cc700e337898bd96c31d80f13d54b2babe1e/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/d262a070a733aa121df137d56530a25ccb688c41fc7712577d303709c52695f3/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/dcd9f41d3cd6d4c718598ed92c49c800971ec12f08cfce46c50e4fd361f6790e/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/682df6e7c58a2875178d3d707127de335c9df9720a6950efbc89f28f388cb27c/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/1c462499bd3b74c1d491ffb94f25e255241b69828dbc1c76470f0b5826615c43/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-63:{mountpoint:/var/lib/containers/storage/overlay/3702d5b663044f35c20c8771a59f028994b9b0563e20bf7fa1ac2fd3e47ae221/merged major:0 minor:63 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/a16e92c55b438b91f49480c6c15091f09d38992fd5ac28e9c0d01e3b01709b8b/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/fdeb17bf8c36c614211951aa1b088fbb2a83abe3ee4fa1f43b06b7cfa7087712/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/d2f0c05353088d371730a2dc6fe52388422dc3122a23dbdc22ccd967165853c6/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/842507c9a2c650e104a40d439c1fc73976acd02c5f6272e675af40752a85b832/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/873bba7e65da6738d7ee4a7b95160ee11c1e151d6bf09577300dae728ecee594/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0146a03fd32baa418bb90b6f90a4facfcde46f2f45d15dfdb76bef86af216148/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/25117a07ed0f996e65d5e6f038902617629fa3365299fc5f153070603abaf926/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/3dd431a47896ba35ca00565da809b0c3b6496454544c9080c77da83a25ea45a4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c314c0d1cbc479d4b8ad75d3eb600651242cf4b970b9fa72676de0228922733b/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 20:29:51.997800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.997372 5932 manager.go:217] Machine: {Timestamp:2026-11-03 20:29:51.996909793 +0000 UTC m=+0.062151258 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101576 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21a0d4b581f7738a2eace6648a1226 SystemUUID:ec21a0d4-b581-f773-8a2e-ace6648a1226 BootID:3fbb41f7-5f41-4e22-8d8c-2e63bea9c03c Filesystems:[{Device:/var/lib/kubelet/pods/536dddb8-cc7a-4b68-a591-7a19d1431afe/volumes/kubernetes.io~projected/kube-api-access-qnfbc DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bbdae4d-8429-4f49-8d31-fe00df3d0701/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f826a9ad04c07b751aff3be97e186fbe31be9152984ce50ede23bb27e9be1f7d/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4895c8-b425-4009-a06b-47f1d51b8784/volumes/kubernetes.io~secret/cluster-monitoring-operator-tls DeviceMajor:0 DeviceMinor:472 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abd8898e1121982a3158ca9931bd6e60f65dcbbb4aba591d873563e426925ca6/userdata/shm DeviceMajor:0 DeviceMinor:476 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:486 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b9c5c799c44e774b3715d51c23ebd24f8a9fe90cac43d3fb0b3258b49276881/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3086db43-8d5a-4196-ab9e-8c7f7832c232/volumes/kubernetes.io~projected/kube-api-access-jwmm8 DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e44f243f-ff0c-4e9a-9a91-b0079c2667b6/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40e2bf21bbff4a94b8fab77e6be0b030442a05330cd4a5c63226467d332e4daf/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5867734bd3cd23c46cf38d7c0e091a1526bb39adcd163a29b4e90e6a29ac79d7/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1546444eef1b2373dc96f9d8fc4b85b42a9483c4024fbc18c3126140c4c1a6f0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-123 DeviceMajor:0 DeviceMinor:123 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e8b4791d1b668bf11d41bdae893b1c2c55e9c3e9040b0d46ce827bbba410ab3/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4895c8-b425-4009-a06b-47f1d51b8784/volumes/kubernetes.io~projected/kube-api-access-8lcbr DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aac3b6b3-61bd-4ee2-b116-61fb0a786896/volumes/kubernetes.io~secret/samples-operator-tls DeviceMajor:0 DeviceMinor:380 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/52568b14-d428-42c1-af42-2e15763fc1b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:451 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcfffc236ff4c616c2373442f45d57a5b0f5e26b840df411bf49d832eeb3a944/userdata/shm DeviceMajor:0 DeviceMinor:494 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/kube-api-access-9f59t DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/052206aa44fbd4f6bbc4702ccb8aa758933b7055ebf6888f113e87c521336d47/userdata/shm DeviceMajor:0 DeviceMinor:478 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18ea77e06f7f7cb9ef311474e3e3fa79654b1a983237738686ab5ccb846c26e7/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/98667931-42d6-42d2-b335-57b2d19d7dfc/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01a555f85d5ab608591f433f4434f7f23cf9b79d9fd51773fac686985bc5acac/userdata/shm DeviceMajor:0 DeviceMinor:389 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-461 DeviceMajor:0 DeviceMinor:461 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea833ecb-ee75-45fd-b689-2b7c0c9a711c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52568b14-d428-42c1-af42-2e15763fc1b3/volumes/kubernetes.io~projected/kube-api-access-smpv4 DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/kube-api-access-qdwdc DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87bd1dcb8ab162396c1d7f2f32d9501adbeb4aa2c50c4fabe9944bc3a2e06f50/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/093b3fa970ad305da00cbb434365c1b3bfbe2da80a8faaf908a1434b2f88a1f5/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc2fe1048fb1fd4aabc4100dcea6e5998cef708ce12d39f05f221aa06bbd4744/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d06f6e8f2b7b7b404682de7397dc3517fccb5bf7f297aa4e31b37574c6e55035/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eee402df-6e0a-4cac-a025-b981a2ee6329/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba8c80837a3bd84d3866cb75dfbc4a2100b258acd6efb759af8645145f563b6d/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:370 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bc681f0-27c6-4cad-8bf6-82835e009976/volumes/kubernetes.io~secret/networking-console-plugin-cert DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ab99b87-827c-4aba-a47e-b86a847ca383/volumes/kubernetes.io~projected/kube-api-access-d942d DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee2d2ede6ca3c7e3a4dd078322b099f61bd48eec44b3229b3ab5104642cdfc97/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a7b21223-f0b4-492b-8856-233fa4b1cd58/volumes/kubernetes.io~projected/kube-api-access-9t97p DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3086db43-8d5a-4196-ab9e-8c7f7832c232/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bfa12a04dbcdc5ce6e7748b5660b6fb62d15971dda6c48a9dda8e75384877b7/userdata/shm DeviceMajor:0 DeviceMinor:393 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7a59cf1-454d-4019-9e05-0108e38b8725/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ea833ecb-ee75-45fd-b689-2b7c0c9a711c/volumes/kubernetes.io~projected/kube-api-access-swnlx DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7a59cf1-454d-4019-9e05-0108e38b8725/volumes/kubernetes.io~projected/kube-api-access-b9skt DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75e6612240f95ce8f7e0d4e8b172bf8def05d8fd4e433623681a3a985814a5b3/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f42c39baf75c158432eb5fb27954f377cee26b00ba47c6a392df9e5126141f28/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab15921-e819-4567-af4a-9d3684c054ae/volumes/kubernetes.io~projected/kube-api-access-bwdln DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe95bd0d-a5b5-44db-97ae-d75a6860a71a/volumes/kubernetes.io~projected/kube-api-access-dlz94 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-202 DeviceMajor:0 DeviceMinor:202 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e44f243f-ff0c-4e9a-9a91-b0079c2667b6/volumes/kubernetes.io~projected/kube-api-access-6pkmm DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b91b91f6-e725-4d7b-b472-9e76650979ab/volumes/kubernetes.io~projected/kube-api-access-kvqhp DeviceMajor:0 DeviceMinor:353 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7/volumes/kubernetes.io~projected/kube-api-access-vtlbn DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68ade7d85a9824584cb99747ebb3397bb991308c8ee9cfefe07e95c11c847811/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff18aebf-b9af-4296-aaad-5d24fd9304ea/volumes/kubernetes.io~projected/kube-api-access-kv629 DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4ddc7ad6ce46409b9a2462cdb4e5c4c1537c78af796d726c71181c772fc190e/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4/volumes/kubernetes.io~projected/kube-api-access-z2rc9 DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98667931-42d6-42d2-b335-57b2d19d7dfc/volumes/kubernetes.io~projected/kube-api-access-p49dx DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e448fa5-814c-427f-854e-02708388f82a/volumes/kubernetes.io~projected/kube-api-access-j877w DeviceMajor:0 DeviceMinor:218 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aac3b6b3-61bd-4ee2-b116-61fb0a786896/volumes/kubernetes.io~projected/kube-api-access-l82xf DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/938f03c8dfc77da966ea41db99d0fcc937fffa8149607f8e7442a86cf9683f88/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07500525b8c01848f2003872296bdaa83810b3f1866778c3d472b00e56891af3/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volumes/kubernetes.io~projected/kube-api-access-pbpnq DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3bbdae4d-8429-4f49-8d31-fe00df3d0701/volumes/kubernetes.io~projected/kube-api-access-j5l82 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f53023301e45a1d74a5ebb8dc3bfe8ac342a1396581483625b6d27eadf59ade9/userdata/shm DeviceMajor:0 DeviceMinor:348 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914/userdata/shm DeviceMajor:0 DeviceMinor:401 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3998a8c3c3bf322ba29418508543fb9bc2d8d6393a6c768d1bfc6dc5f8ad8d93/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c947407f-1cc6-4c01-997e-99e19a3c139c/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:491 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4b5314e5149d0e3699279052fecbb1c70388142640c2cd30852143eae17aa1b/userdata/shm DeviceMajor:0 DeviceMinor:492 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-63 DeviceMajor:0 DeviceMinor:63 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a995cca6-542f-406e-a938-aa58715eac9a/volumes/kubernetes.io~projected/kube-api-access-4q4l5 DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a/volumes/kubernetes.io~projected/kube-api-access-ffww8 DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b30d352ff73c029527f63e4a2415c5922c763aeda4ef77005727df3d42cd966a/userdata/shm DeviceMajor:0 DeviceMinor:455 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/acf17209-903f-4a4e-9baf-908e3ab9afae/volumes/kubernetes.io~projected/kube-api-access-wzs87 DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/311c1fbb-32b5-4d7d-934b-28b68c8d48cb/volumes/kubernetes.io~projected/kube-api-access-9qz8f DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e448fa5-814c-427f-854e-02708388f82a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/02bb73c6-e6e2-4d77-8458-20ba459b831a/volumes/kubernetes.io~projected/kube-api-access-qxxd9 DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/157e7b6592d84ad433fdf187f51d4422804f5cd29be7b904d25392e914a412c5/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:368 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750/volumes/kubernetes.io~projected/kube-api-access-6htdx DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5ca1597-5b32-4231-97cd-e041e4ff6cd4/volumes/kubernetes.io~projected/kube-api-access-nfj4h DeviceMajor:0 DeviceMinor:226 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1b337cfed0a5780d7dab584680ed75c9ba3cf8c67549be5f9aebc1648f8376a/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f34740ae-e26e-4188-a405-4da96b57553e/volumes/kubernetes.io~projected/kube-api-access-rzg7d DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7b8419bd-5051-4126-9569-b6344e49c66e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3/volumes/kubernetes.io~projected/kube-api-access-zgm99 DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01a555f85d5ab60 MacAddress:e2:f4:c2:bb:bc:8f Speed:10000 Mtu:8901} {Name:052206aa44fbd4f MacAddress:1e:a1:aa:e1:29:02 Speed:10000 Mtu:8901} {Name:07500525b8c0184 MacAddress:7a:7d:f1:73:38:19 Speed:10000 Mtu:8901} {Name:093b3fa970ad305 MacAddress:1a:58:76:aa:84:3e Speed:10000 Mtu:8901} {Name:157e7b6592d84ad MacAddress:aa:06:f9:95:73:8f Speed:10000 Mtu:8901} {Name:1bfa12a04dbcdc5 MacAddress:ce:bb:56:2c:ee:60 Speed:10000 Mtu:8901} {Name:2b9c5c799c44e77 MacAddress:4e:a6:af:ac:43:39 Speed:10000 Mtu:8901} {Name:3e8b4791d1b668b MacAddress:d6:33:7f:b4:c8:61 Speed:10000 Mtu:8901} {Name:75e6612240f95ce MacAddress:9e:18:46:f3:ff:81 Speed:10000 Mtu:8901} {Name:7f5f4c854dbe334 MacAddress:be:a6:4c:5f:2a:df Speed:10000 Mtu:8901} {Name:87bd1dcb8ab1623 MacAddress:f2:08:70:a3:6b:0d Speed:10000 Mtu:8901} {Name:938f03c8dfc77da MacAddress:0a:62:63:2e:af:3b Speed:10000 Mtu:8901} {Name:a1b337cfed0a578 MacAddress:5a:54:5a:8f:cc:02 Speed:10000 Mtu:8901} {Name:a4ddc7ad6ce4640 MacAddress:5e:de:4b:8e:66:33 Speed:10000 Mtu:8901} {Name:abd8898e1121982 MacAddress:8e:4e:1c:66:36:05 Speed:10000 Mtu:8901} {Name:b30d352ff73c029 MacAddress:3e:9a:cd:08:45:e7 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:fa:ef:29:0b Speed:0 Mtu:9001} {Name:br-int MacAddress:96:34:fc:38:c3:9f Speed:0 Mtu:8901} {Name:c5429167018357a MacAddress:7e:7f:2f:c5:76:d3 Speed:10000 Mtu:8901} {Name:ca093a850baa382 MacAddress:f6:59:1f:ad:6f:ae Speed:10000 Mtu:8901} {Name:d06f6e8f2b7b7b4 MacAddress:92:95:18:be:09:d4 Speed:10000 Mtu:8901} {Name:dcfffc236ff4c61 MacAddress:6a:2d:29:6c:9f:d9 Speed:10000 Mtu:8901} {Name:ee2d2ede6ca3c7e MacAddress:c2:cf:10:2e:71:fa Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:fa:ef:29:0b Speed:0 Mtu:9001} {Name:f42c39baf75c158 MacAddress:62:3b:20:44:8a:9f Speed:10000 Mtu:8901} {Name:f4b5314e5149d0e MacAddress:a2:db:de:7b:d2:ea Speed:10000 Mtu:8901} {Name:f53023301e45a1d MacAddress:1a:1d:5c:64:4f:9d Speed:10000 Mtu:8901} {Name:f826a9ad04c07b7 MacAddress:9a:c4:15:85:b5:66 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:7a:4c:31:6c:5b:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:3f:8d:62:fa:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 20:29:51.998051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.997804 5932 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 20:29:51.998051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.997848 5932 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 20:29:51.998128 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998107 5932 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 20:29:51.998252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998128 5932 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-52.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 20:29:51.998293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998254 5932 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 20:29:51.998293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998262 5932 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 20:29:51.998293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998274 5932 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:29:51.998293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998292 5932 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 20:29:51.998399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998310 5932 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 20:29:51.998470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998460 5932 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 20:29:51.998586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998577 5932 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 20:29:51.998617 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998612 5932 kubelet.go:493] "Attempting to sync node with API server" Nov 03 20:29:51.998658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998636 5932 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 20:29:51.998658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998648 5932 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 20:29:51.998658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998657 5932 kubelet.go:405] "Adding apiserver pod source" Nov 03 20:29:51.998764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.998665 5932 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 20:29:51.999527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.999507 5932 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:29:51.999527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.999523 5932 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:29:52.000406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:51.999960 5932 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 20:29:52.003835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.003818 5932 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 20:29:52.003835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.003832 5932 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 20:29:52.003954 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.003943 5932 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 20:29:52.004147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004130 5932 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 20:29:52.004183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004160 5932 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 20:29:52.004269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004259 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 20:29:52.004299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004274 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 20:29:52.004299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004280 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 20:29:52.004299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004286 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 20:29:52.004299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004293 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004302 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004312 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004321 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004328 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004333 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004342 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004351 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004377 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 20:29:52.004408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.004385 5932 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 20:29:52.007814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.007799 5932 server.go:1263] "Started kubelet" Nov 03 20:29:52.007908 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.007875 5932 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 20:29:52.007961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.007894 5932 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 20:29:52.008009 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.007988 5932 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 20:29:52.008407 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.008388 5932 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 20:29:52.008936 ip-10-0-138-52 systemd[1]: Started Kubernetes Kubelet. Nov 03 20:29:52.009248 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.009231 5932 server.go:317] "Adding debug handlers to kubelet server" Nov 03 20:29:52.011587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011570 5932 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 20:29:52.011735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011585 5932 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 20:29:52.011735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011716 5932 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 20:29:52.011735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011713 5932 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 20:23:41 +0000 UTC" deadline="2028-05-19 14:15:00.717826416 +0000 UTC" Nov 03 20:29:52.011735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011725 5932 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 20:29:52.011735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011730 5932 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13505h45m8.706099337s" Nov 03 20:29:52.012084 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.011769 5932 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" Nov 03 20:29:52.012084 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.011806 5932 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 20:29:52.014897 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.013288 5932 factory.go:55] Registering systemd factory Nov 03 20:29:52.014897 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.013309 5932 factory.go:223] Registration of the systemd container factory successfully Nov 03 20:29:52.016207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.016183 5932 factory.go:153] Registering CRI-O factory Nov 03 20:29:52.016207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.016209 5932 factory.go:223] Registration of the crio container factory successfully Nov 03 20:29:52.016340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.016262 5932 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 20:29:52.016340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.016279 5932 factory.go:103] Registering Raw factory Nov 03 20:29:52.016340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.016294 5932 manager.go:1199] Started watching for new ooms in manager Nov 03 20:29:52.018946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.016725 5932 manager.go:319] Starting recovery of all containers Nov 03 20:29:52.018946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.018660 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" volumeName="kubernetes.io/projected/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-kube-api-access-9qz8f" seLinuxMountContext="" Nov 03 20:29:52.018946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.018802 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bbdae4d-8429-4f49-8d31-fe00df3d0701" volumeName="kubernetes.io/projected/3bbdae4d-8429-4f49-8d31-fe00df3d0701-kube-api-access-j5l82" seLinuxMountContext="" Nov 03 20:29:52.018946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.018827 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token" seLinuxMountContext="" Nov 03 20:29:52.019127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.018880 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls" seLinuxMountContext="" Nov 03 20:29:52.019127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019017 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b91b91f6-e725-4d7b-b472-9e76650979ab" volumeName="kubernetes.io/projected/b91b91f6-e725-4d7b-b472-9e76650979ab-kube-api-access-kvqhp" seLinuxMountContext="" Nov 03 20:29:52.019475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019031 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" volumeName="kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-binary-copy" seLinuxMountContext="" Nov 03 20:29:52.019576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019486 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca" seLinuxMountContext="" Nov 03 20:29:52.019576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019504 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4895c8-b425-4009-a06b-47f1d51b8784" volumeName="kubernetes.io/configmap/6b4895c8-b425-4009-a06b-47f1d51b8784-telemetry-config" seLinuxMountContext="" Nov 03 20:29:52.019576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019533 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98667931-42d6-42d2-b335-57b2d19d7dfc" volumeName="kubernetes.io/secret/98667931-42d6-42d2-b335-57b2d19d7dfc-serving-cert" seLinuxMountContext="" Nov 03 20:29:52.019576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019547 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a995cca6-542f-406e-a938-aa58715eac9a" volumeName="kubernetes.io/projected/a995cca6-542f-406e-a938-aa58715eac9a-kube-api-access-4q4l5" seLinuxMountContext="" Nov 03 20:29:52.019576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019560 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca" seLinuxMountContext="" Nov 03 20:29:52.019576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019571 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019583 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019596 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3086db43-8d5a-4196-ab9e-8c7f7832c232" volumeName="kubernetes.io/configmap/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-cabundle" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019613 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019639 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019651 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf17209-903f-4a4e-9baf-908e3ab9afae" volumeName="kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-tmp" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019662 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e44f243f-ff0c-4e9a-9a91-b0079c2667b6" volumeName="kubernetes.io/empty-dir/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-tmp-dir" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019684 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" volumeName="kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019696 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe95bd0d-a5b5-44db-97ae-d75a6860a71a" volumeName="kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019710 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab15921-e819-4567-af4a-9d3684c054ae" volumeName="kubernetes.io/projected/2ab15921-e819-4567-af4a-9d3684c054ae-kube-api-access-bwdln" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019722 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" volumeName="kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019733 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" volumeName="kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019747 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5ca1597-5b32-4231-97cd-e041e4ff6cd4" volumeName="kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-stats-auth" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019760 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" volumeName="kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019773 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019786 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4895c8-b425-4009-a06b-47f1d51b8784" volumeName="kubernetes.io/projected/6b4895c8-b425-4009-a06b-47f1d51b8784-kube-api-access-8lcbr" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019798 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b21223-f0b4-492b-8856-233fa4b1cd58" volumeName="kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-daemon-config" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019848 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" volumeName="kubernetes.io/projected/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-kube-api-access-vtlbn" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019862 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff18aebf-b9af-4296-aaad-5d24fd9304ea" volumeName="kubernetes.io/projected/ff18aebf-b9af-4296-aaad-5d24fd9304ea-kube-api-access-kv629" seLinuxMountContext="" Nov 03 20:29:52.019889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019874 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019887 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc681f0-27c6-4cad-8bf6-82835e009976" volumeName="kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019932 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a59cf1-454d-4019-9e05-0108e38b8725" volumeName="kubernetes.io/projected/c7a59cf1-454d-4019-9e05-0108e38b8725-kube-api-access-b9skt" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019945 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a59cf1-454d-4019-9e05-0108e38b8725" volumeName="kubernetes.io/secret/c7a59cf1-454d-4019-9e05-0108e38b8725-serving-cert" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019960 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" volumeName="kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019972 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52568b14-d428-42c1-af42-2e15763fc1b3" volumeName="kubernetes.io/empty-dir/52568b14-d428-42c1-af42-2e15763fc1b3-data-volume" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019984 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf17209-903f-4a4e-9baf-908e3ab9afae" volumeName="kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-tuned" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.019997 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c947407f-1cc6-4c01-997e-99e19a3c139c" volumeName="kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020008 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee402df-6e0a-4cac-a025-b981a2ee6329" volumeName="kubernetes.io/secret/eee402df-6e0a-4cac-a025-b981a2ee6329-agent-certs" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020028 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5ca1597-5b32-4231-97cd-e041e4ff6cd4" volumeName="kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020035 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab15921-e819-4567-af4a-9d3684c054ae" volumeName="kubernetes.io/configmap/2ab15921-e819-4567-af4a-9d3684c054ae-serviceca" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020044 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3086db43-8d5a-4196-ab9e-8c7f7832c232" volumeName="kubernetes.io/secret/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-key" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020053 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020064 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" volumeName="kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020076 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8419bd-5051-4126-9569-b6344e49c66e" volumeName="kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-config" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020086 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8419bd-5051-4126-9569-b6344e49c66e" volumeName="kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-script-lib" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020093 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a59cf1-454d-4019-9e05-0108e38b8725" volumeName="kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-trusted-ca" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020101 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" volumeName="kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-snapshots" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020108 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98667931-42d6-42d2-b335-57b2d19d7dfc" volumeName="kubernetes.io/configmap/98667931-42d6-42d2-b335-57b2d19d7dfc-config" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020115 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3bbdae4d-8429-4f49-8d31-fe00df3d0701" volumeName="kubernetes.io/secret/3bbdae4d-8429-4f49-8d31-fe00df3d0701-cert" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020123 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020131 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="536dddb8-cc7a-4b68-a591-7a19d1431afe" volumeName="kubernetes.io/projected/536dddb8-cc7a-4b68-a591-7a19d1431afe-kube-api-access-qnfbc" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020138 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e448fa5-814c-427f-854e-02708388f82a" volumeName="kubernetes.io/secret/5e448fa5-814c-427f-854e-02708388f82a-serving-cert" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020145 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4895c8-b425-4009-a06b-47f1d51b8784" volumeName="kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020176 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acf17209-903f-4a4e-9baf-908e3ab9afae" volumeName="kubernetes.io/projected/acf17209-903f-4a4e-9baf-908e3ab9afae-kube-api-access-wzs87" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020189 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" volumeName="kubernetes.io/projected/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-kube-api-access-swnlx" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020210 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02bb73c6-e6e2-4d77-8458-20ba459b831a" volumeName="kubernetes.io/projected/02bb73c6-e6e2-4d77-8458-20ba459b831a-kube-api-access-qxxd9" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020222 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bc681f0-27c6-4cad-8bf6-82835e009976" volumeName="kubernetes.io/configmap/5bc681f0-27c6-4cad-8bf6-82835e009976-nginx-conf" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020236 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8419bd-5051-4126-9569-b6344e49c66e" volumeName="kubernetes.io/secret/7b8419bd-5051-4126-9569-b6344e49c66e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020247 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b21223-f0b4-492b-8856-233fa4b1cd58" volumeName="kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-cni-binary-copy" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020258 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020268 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" volumeName="kubernetes.io/secret/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-serving-cert" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020279 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f34740ae-e26e-4188-a405-4da96b57553e" volumeName="kubernetes.io/projected/f34740ae-e26e-4188-a405-4da96b57553e-kube-api-access-rzg7d" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020295 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5ca1597-5b32-4231-97cd-e041e4ff6cd4" volumeName="kubernetes.io/projected/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-kube-api-access-nfj4h" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020305 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52568b14-d428-42c1-af42-2e15763fc1b3" volumeName="kubernetes.io/projected/52568b14-d428-42c1-af42-2e15763fc1b3-kube-api-access-smpv4" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020315 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e448fa5-814c-427f-854e-02708388f82a" volumeName="kubernetes.io/projected/5e448fa5-814c-427f-854e-02708388f82a-kube-api-access-j877w" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020326 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750" volumeName="kubernetes.io/projected/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750-kube-api-access-6htdx" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020337 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ab99b87-827c-4aba-a47e-b86a847ca383" volumeName="kubernetes.io/projected/6ab99b87-827c-4aba-a47e-b86a847ca383-kube-api-access-d942d" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020356 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8419bd-5051-4126-9569-b6344e49c66e" volumeName="kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-env-overrides" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020369 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aac3b6b3-61bd-4ee2-b116-61fb0a786896" volumeName="kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020380 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" volumeName="kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-tmp" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020392 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5ca1597-5b32-4231-97cd-e041e4ff6cd4" volumeName="kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020402 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="536dddb8-cc7a-4b68-a591-7a19d1431afe" volumeName="kubernetes.io/configmap/536dddb8-cc7a-4b68-a591-7a19d1431afe-iptables-alerter-script" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020414 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b8419bd-5051-4126-9569-b6344e49c66e" volumeName="kubernetes.io/projected/7b8419bd-5051-4126-9569-b6344e49c66e-kube-api-access-pbpnq" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020425 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e44f243f-ff0c-4e9a-9a91-b0079c2667b6" volumeName="kubernetes.io/projected/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-kube-api-access-6pkmm" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020436 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5ca1597-5b32-4231-97cd-e041e4ff6cd4" volumeName="kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-default-certificate" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020448 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aac3b6b3-61bd-4ee2-b116-61fb0a786896" volumeName="kubernetes.io/configmap/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-config" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020460 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020472 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e44f243f-ff0c-4e9a-9a91-b0079c2667b6" volumeName="kubernetes.io/configmap/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-config-volume" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020484 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea25f895-6d8c-4bf1-abd4-6e29c4a772e7" volumeName="kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020495 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" volumeName="kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-service-ca-bundle" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020507 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f34740ae-e26e-4188-a405-4da96b57553e" volumeName="kubernetes.io/empty-dir/f34740ae-e26e-4188-a405-4da96b57553e-tmp-dir" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020546 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" volumeName="kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020558 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4" volumeName="kubernetes.io/projected/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4-kube-api-access-z2rc9" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020569 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eee402df-6e0a-4cac-a025-b981a2ee6329" volumeName="kubernetes.io/configmap/eee402df-6e0a-4cac-a025-b981a2ee6329-konnectivity-ca" seLinuxMountContext="" Nov 03 20:29:52.020671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020592 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3086db43-8d5a-4196-ab9e-8c7f7832c232" volumeName="kubernetes.io/projected/3086db43-8d5a-4196-ab9e-8c7f7832c232-kube-api-access-jwmm8" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020614 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4382cc48-e16d-4e80-85af-ff5f5d715c7a" volumeName="kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020731 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52568b14-d428-42c1-af42-2e15763fc1b3" volumeName="kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020749 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e448fa5-814c-427f-854e-02708388f82a" volumeName="kubernetes.io/configmap/5e448fa5-814c-427f-854e-02708388f82a-config" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020769 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f0d106f-4b85-48cf-8c77-6b4966cfd1f3" volumeName="kubernetes.io/projected/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kube-api-access-zgm99" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020785 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98667931-42d6-42d2-b335-57b2d19d7dfc" volumeName="kubernetes.io/projected/98667931-42d6-42d2-b335-57b2d19d7dfc-kube-api-access-p49dx" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020797 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abc63a7c-b9ed-4269-90aa-49a71a94d932" volumeName="kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020809 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aac3b6b3-61bd-4ee2-b116-61fb0a786896" volumeName="kubernetes.io/projected/aac3b6b3-61bd-4ee2-b116-61fb0a786896-kube-api-access-l82xf" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020828 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52568b14-d428-42c1-af42-2e15763fc1b3" volumeName="kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020841 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7b21223-f0b4-492b-8856-233fa4b1cd58" volumeName="kubernetes.io/projected/a7b21223-f0b4-492b-8856-233fa4b1cd58-kube-api-access-9t97p" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020853 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7a59cf1-454d-4019-9e05-0108e38b8725" volumeName="kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-config" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020864 5932 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e44f243f-ff0c-4e9a-9a91-b0079c2667b6" volumeName="kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls" seLinuxMountContext="" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020875 5932 reconstruct.go:97] "Volume reconstruction finished" Nov 03 20:29:52.022900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.020882 5932 reconciler.go:29] "Reconciler: start to sync state" Nov 03 20:29:52.028237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.028216 5932 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 20:29:52.033405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.033384 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:29:52.033554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.033521 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:29:52.043445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.043416 5932 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 20:29:52.043445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.043441 5932 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 20:29:52.043588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.043462 5932 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 20:29:52.043588 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.043508 5932 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 20:29:52.059642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.059600 5932 generic.go:356] "Generic (PLEG): container finished" podID="ca837b7a3905083ff9db7ad20c4a2d6d" containerID="76fee7a26009e19fa86de139f602f20daf7925afc1fbe34d56d5674145c131bd" exitCode=0 Nov 03 20:29:52.097874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.097827 5932 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="e13b245d5c2e02cb8e94d879f3fc89b4a00087dcf4c56c987509ee1e8c398793" exitCode=0 Nov 03 20:29:52.097874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.097874 5932 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="312c9b98175a36960fe4c92de1f2830a79bdcbda03fb87dc5ac4c6f18c48f7c3" exitCode=0 Nov 03 20:29:52.098064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.097889 5932 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="c25a652b56db1372a4c20b50b15b545bf3213413138834ade5e93a08298297ac" exitCode=0 Nov 03 20:29:52.098064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.097905 5932 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="f78a0810cb7e7548d9f257237559b48eac86874eb7267af1c798ebba0fe7e4ba" exitCode=0 Nov 03 20:29:52.098064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.097918 5932 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="3de40bd4ecc5acccfaf5591ef5f9f610eeba4e659874df731f4ad3a7c04197b3" exitCode=0 Nov 03 20:29:52.098064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.097931 5932 generic.go:356] "Generic (PLEG): container finished" podID="311c1fbb-32b5-4d7d-934b-28b68c8d48cb" containerID="9c5d74b446f78eae985143f2d46d78749c757a6cee42cf85783dadef3bcec1bc" exitCode=0 Nov 03 20:29:52.111834 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.111809 5932 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" Nov 03 20:29:52.123075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.123013 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:29:52.123075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.123038 5932 generic.go:356] "Generic (PLEG): container finished" podID="c7a59cf1-454d-4019-9e05-0108e38b8725" containerID="f219ed3d07565cc2c9d65c1dfaf7ab5cbee956d804542905217e557e6470910e" exitCode=255 Nov 03 20:29:52.140577 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.140546 5932 manager.go:324] Recovery completed Nov 03 20:29:52.143661 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.143642 5932 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 20:29:52.164546 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.164519 5932 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:29:52.166482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.166463 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:29:52.166578 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.166499 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:29:52.166578 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.166515 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:29:52.169191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169171 5932 cpu_manager.go:226] "Starting" policy="none" Nov 03 20:29:52.169191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169187 5932 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 20:29:52.169333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169211 5932 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 20:29:52.169333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169190 5932 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:29:52.169448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169375 5932 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 20:29:52.169448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169390 5932 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 20:29:52.169448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169418 5932 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 20:29:52.169448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169426 5932 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 20:29:52.169448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169434 5932 policy_none.go:50] "Start" Nov 03 20:29:52.169448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169447 5932 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 20:29:52.169753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169461 5932 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 20:29:52.169753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169669 5932 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 20:29:52.169753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169681 5932 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 20:29:52.169753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.169690 5932 policy_none.go:44] "Start" Nov 03 20:29:52.170164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.170149 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:29:52.170228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.170176 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:29:52.170228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.170189 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:29:52.174691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.174666 5932 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 20:29:52.178046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178033 5932 manager.go:341] "Starting Device Plugin manager" Nov 03 20:29:52.178132 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.178066 5932 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 20:29:52.178132 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178075 5932 server.go:87] "Starting device plugin registration server" Nov 03 20:29:52.178327 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178314 5932 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 20:29:52.178390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178331 5932 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 20:29:52.178438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178406 5932 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 20:29:52.178590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178545 5932 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 20:29:52.179316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178598 5932 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 20:29:52.179316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178606 5932 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 20:29:52.179316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.178651 5932 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T20:29:52.178588144Z","UUID":"37491254-7f92-44f4-9a53-c026975a6a91","Handler":null,"Name":"","Endpoint":""} Nov 03 20:29:52.181138 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.181118 5932 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 20:29:52.181235 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.181175 5932 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-52.us-west-2.compute.internal\" not found" Nov 03 20:29:52.181235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.181178 5932 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 20:29:52.181235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.181233 5932 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 20:29:52.197702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.197677 5932 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:29:52.197799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.197782 5932 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:29:52.197849 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.197833 5932 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 20:29:52.200470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.200454 5932 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:29:52.244164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.244139 5932 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 20:29:52.279177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.279161 5932 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 20:29:52.280070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.280050 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 20:29:52.280158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.280080 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 20:29:52.280158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.280092 5932 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-52.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 20:29:52.280158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.280138 5932 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.344003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.343972 5932 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal"] Nov 03 20:29:52.344609 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344591 5932 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.344687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344573 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"ca837b7a3905083ff9db7ad20c4a2d6d","Type":"ContainerStarted","Data":"b9a19652f447e015c201c12ab2c344164ab8d87562eef9fe26a4bebe592cc9e3"} Nov 03 20:29:52.344734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344717 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"ca837b7a3905083ff9db7ad20c4a2d6d","Type":"ContainerDied","Data":"76fee7a26009e19fa86de139f602f20daf7925afc1fbe34d56d5674145c131bd"} Nov 03 20:29:52.344765 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344746 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"ca837b7a3905083ff9db7ad20c4a2d6d","Type":"ContainerStarted","Data":"2873267e6f5b21eb382c0f8d5ca90b4a8eb34034370c4b13acdd00d7cf2e841a"} Nov 03 20:29:52.344855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344845 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"d46efce11f8f6dc55307823b4dc3bdd4","Type":"ContainerStarted","Data":"7b7a56108c6463f30cd6abeb1ff6114cc927aeb6b43e7d948ef5e293c0f38529"} Nov 03 20:29:52.344886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344862 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" event={"ID":"d46efce11f8f6dc55307823b4dc3bdd4","Type":"ContainerStarted","Data":"8c5eaf6a9fea90ea3b4ceea177fd9c3bc896090168c0dd2f263dd9105f55caf9"} Nov 03 20:29:52.344886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.344875 5932 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.382080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.382063 5932 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.382138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.382114 5932 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.401694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.401677 5932 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:29:52.401776 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.401711 5932 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.402313 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.402302 5932 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:29:52.402367 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:52.402324 5932 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.499058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.499032 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.499172 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.499062 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d46efce11f8f6dc55307823b4dc3bdd4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"d46efce11f8f6dc55307823b4dc3bdd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.499172 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.499078 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.599702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.599668 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.599863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.599708 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.599863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.599734 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d46efce11f8f6dc55307823b4dc3bdd4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"d46efce11f8f6dc55307823b4dc3bdd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.599863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.599779 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d46efce11f8f6dc55307823b4dc3bdd4-config\") pod \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"d46efce11f8f6dc55307823b4dc3bdd4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.599863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.599780 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.599863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.599780 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ca837b7a3905083ff9db7ad20c4a2d6d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" (UID: \"ca837b7a3905083ff9db7ad20c4a2d6d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:52.999220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:52.999175 5932 apiserver.go:52] "Watching apiserver" Nov 03 20:29:53.096508 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.096485 5932 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 20:29:53.097194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.097171 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-627kh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2","openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d","kube-system/kas-connection-checker-685895c7c9-8bznz","kube-system/kas-connection-checker-685895c7c9-48l5q","openshift-cluster-node-tuning-operator/tuned-2fhq4","openshift-ovn-kubernetes/ovnkube-node-w6gnn","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt","openshift-image-registry/image-registry-7b5c54f4d6-pwkct","kube-system/konnectivity-agent-l5fnw","openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh","openshift-service-ca/service-ca-5fffc97bdf-x5fbx","openshift-ingress/router-default-76f9745bc9-fxgzq","openshift-image-registry/node-ca-2j4jr","openshift-network-console/networking-console-plugin-86c478bb88-2b69b","openshift-network-diagnostics/network-check-target-9x8q2","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b","openshift-dns/dns-default-sd2tb","openshift-multus/multus-qxfrx","kube-system/global-pull-secret-syncer-5khcq","openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-bw59n","openshift-dns/node-resolver-cwd62","openshift-image-registry/image-registry-595485878c-mqdnk","kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-8rvtd","openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7","openshift-multus/network-metrics-daemon-vfs2r","kube-system/kas-connection-checker-685895c7c9-k2tz8","openshift-insights/insights-runtime-extractor-dpsrk","openshift-network-operator/iptables-alerter-qkfwr","openshift-console-operator/console-operator-7cd56b8dbc-ljjts","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7","openshift-insights/insights-operator-7c4986d449-zd9mv","openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:29:53.127797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.127775 5932 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:53.127930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.127918 5932 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:53.146514 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.146495 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.146707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.146673 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.146767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.146722 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.146801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.146767 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.147540 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.147503 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-pcxgh\"" Nov 03 20:29:53.147813 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.147801 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 20:29:53.147849 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.147839 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 20:29:53.148049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148034 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-n968g\"" Nov 03 20:29:53.148378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148362 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.148430 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148384 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.148668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148586 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.148818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148797 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.148925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148834 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.148993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148929 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.148993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.148936 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.149090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.149008 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.149132 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.149116 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 20:29:53.149294 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.149278 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-9fv67\"" Nov 03 20:29:53.149359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.149328 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 20:29:53.149855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.149837 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7sh8x\"" Nov 03 20:29:53.150524 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.150502 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 20:29:53.150637 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.150580 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 20:29:53.150839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.150817 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-v9mbf\"" Nov 03 20:29:53.150901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.150888 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.151191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.151173 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 20:29:53.151445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.151426 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gpslm\"" Nov 03 20:29:53.151584 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.151561 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 20:29:53.153210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.153188 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 20:29:53.203115 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203091 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-default-certificate\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.203199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203121 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-cnibin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.203199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203145 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-registration-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.203199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203162 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-device-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.203199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203180 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.203353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203278 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.203353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203319 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-host\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.203353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203328 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.203353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203338 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b4895c8-b425-4009-a06b-47f1d51b8784-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:53.203505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203493 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdln\" (UniqueName: \"kubernetes.io/projected/2ab15921-e819-4567-af4a-9d3684c054ae-kube-api-access-bwdln\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.203555 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203518 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6b4895c8-b425-4009-a06b-47f1d51b8784-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:53.203555 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203532 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.203687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203660 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/536dddb8-cc7a-4b68-a591-7a19d1431afe-iptables-alerter-script\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.203746 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203690 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-tmp\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.203797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203745 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-var-lib-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.203797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203753 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-tmp\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.203797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203764 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-etc-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.203797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203793 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.203913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203815 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.203913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203840 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98667931-42d6-42d2-b335-57b2d19d7dfc-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:53.203913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203902 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlbn\" (UniqueName: \"kubernetes.io/projected/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-kube-api-access-vtlbn\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:53.204010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203920 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-conf\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.204010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203936 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-conf-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.204010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203952 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:53.204010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.203968 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:53.204168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204011 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.204168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204046 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9skt\" (UniqueName: \"kubernetes.io/projected/c7a59cf1-454d-4019-9e05-0108e38b8725-kube-api-access-b9skt\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.204168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204072 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.204168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204085 5932 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:29:53.204168 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:53.204131 5932 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:53.204168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204138 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:53.204372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204191 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/98667931-42d6-42d2-b335-57b2d19d7dfc-serving-cert\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:53.204372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204257 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-var-lib-kubelet\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.204372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204296 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-system-cni-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.204372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204309 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6b4895c8-b425-4009-a06b-47f1d51b8784-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:53.204372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204322 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-lib-modules\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.204527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204399 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-config\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.204527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204436 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-trusted-ca\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.204527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204512 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e448fa5-814c-427f-854e-02708388f82a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:53.204653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204544 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-socket-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.204653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204572 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-config\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.204653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204572 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8419bd-5051-4126-9569-b6344e49c66e-ovn-node-metrics-cert\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.204810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204672 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-cni-binary-copy\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.204810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204732 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-log-socket\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.204810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204766 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-netd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.204810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204772 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7a59cf1-454d-4019-9e05-0108e38b8725-trusted-ca\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.204810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204741 5932 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 20:29:53.204960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204811 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-config\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.204960 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:29:53.204835 5932 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-52.us-west-2.compute.internal" Nov 03 20:29:53.204960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204852 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysconfig\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.204960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204877 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-systemd\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.204960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204906 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.204960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.204924 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-multus-certs\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.205125 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205038 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-modprobe-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.205125 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205089 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.205182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205125 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f34740ae-e26e-4188-a405-4da96b57553e-hosts-file\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.205182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205148 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d942d\" (UniqueName: \"kubernetes.io/projected/6ab99b87-827c-4aba-a47e-b86a847ca383-kube-api-access-d942d\") pod \"volume-data-source-validator-58c9f5f46d-b74gt\" (UID: \"6ab99b87-827c-4aba-a47e-b86a847ca383\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:53.205182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205165 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6htdx\" (UniqueName: \"kubernetes.io/projected/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750-kube-api-access-6htdx\") pod \"kas-connection-checker-685895c7c9-k2tz8\" (UID: \"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750\") " pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:53.205320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205186 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98667931-42d6-42d2-b335-57b2d19d7dfc-config\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:53.205320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205211 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:53.205320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205250 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpnq\" (UniqueName: \"kubernetes.io/projected/7b8419bd-5051-4126-9569-b6344e49c66e-kube-api-access-pbpnq\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.205320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205283 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-kubernetes\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.205320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205311 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-slash\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.205512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205335 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-dbus\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.205512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205460 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/98667931-42d6-42d2-b335-57b2d19d7dfc-config\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:53.205512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205467 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ab15921-e819-4567-af4a-9d3684c054ae-serviceca\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.205597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205509 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eee402df-6e0a-4cac-a025-b981a2ee6329-konnectivity-ca\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:53.205597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205537 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bc681f0-27c6-4cad-8bf6-82835e009976-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:53.205597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205563 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49dx\" (UniqueName: \"kubernetes.io/projected/98667931-42d6-42d2-b335-57b2d19d7dfc-kube-api-access-p49dx\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:53.205597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205592 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-socket-dir-parent\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.205728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205611 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-service-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.205728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205617 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-daemon-config\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.205728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205674 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-netns\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.205728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205722 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.205842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205764 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-tmp\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.205842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205798 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-stats-auth\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.205842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205816 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-system-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.205842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205824 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-tmp\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.205842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205835 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-systemd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.205976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205899 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.205976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205926 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnfbc\" (UniqueName: \"kubernetes.io/projected/536dddb8-cc7a-4b68-a591-7a19d1431afe-kube-api-access-qnfbc\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.205976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205947 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j877w\" (UniqueName: \"kubernetes.io/projected/5e448fa5-814c-427f-854e-02708388f82a-kube-api-access-j877w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:53.205976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205963 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205979 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/536dddb8-cc7a-4b68-a591-7a19d1431afe-host-slash\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.205995 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-snapshots\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206015 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e448fa5-814c-427f-854e-02708388f82a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206045 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-etc-kubernetes\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206080 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206106 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34740ae-e26e-4188-a405-4da96b57553e-tmp-dir\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.206129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206120 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-snapshots\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206132 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfj4h\" (UniqueName: \"kubernetes.io/projected/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-kube-api-access-nfj4h\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206158 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-sys-fs\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206185 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-systemd-units\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206212 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f34740ae-e26e-4188-a405-4da96b57553e-tmp-dir\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206218 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-env-overrides\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206244 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cnibin\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206261 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzg7d\" (UniqueName: \"kubernetes.io/projected/f34740ae-e26e-4188-a405-4da96b57553e-kube-api-access-rzg7d\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206279 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnlx\" (UniqueName: \"kubernetes.io/projected/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-kube-api-access-swnlx\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.206392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206399 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-hostroot\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206430 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-run\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206455 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-tuned\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206492 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rc9\" (UniqueName: \"kubernetes.io/projected/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4-kube-api-access-z2rc9\") pod \"network-check-source-6f78d59f99-vmfmg\" (UID: \"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206509 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206530 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-sys\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206583 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206650 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-multus\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206677 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-tuned\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206681 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206721 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ab15921-e819-4567-af4a-9d3684c054ae-host\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206752 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t97p\" (UniqueName: \"kubernetes.io/projected/a7b21223-f0b4-492b-8856-233fa4b1cd58-kube-api-access-9t97p\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.206806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206782 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-kubelet\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206844 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-node-log\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206890 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qz8f\" (UniqueName: \"kubernetes.io/projected/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-kube-api-access-9qz8f\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.206939 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxxd9\" (UniqueName: \"kubernetes.io/projected/02bb73c6-e6e2-4d77-8458-20ba459b831a-kube-api-access-qxxd9\") pod \"kas-connection-checker-685895c7c9-8bznz\" (UID: \"02bb73c6-e6e2-4d77-8458-20ba459b831a\") " pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207046 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-k8s-cni-cncf-io\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207071 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-netns\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207103 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207132 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-ovn\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207165 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-kubelet-config\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207202 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207260 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:53.207371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207367 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82xf\" (UniqueName: \"kubernetes.io/projected/aac3b6b3-61bd-4ee2-b116-61fb0a786896-kube-api-access-l82xf\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207386 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207402 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q4l5\" (UniqueName: \"kubernetes.io/projected/a995cca6-542f-406e-a938-aa58715eac9a-kube-api-access-4q4l5\") pod \"kas-connection-checker-685895c7c9-48l5q\" (UID: \"a995cca6-542f-406e-a938-aa58715eac9a\") " pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207420 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207460 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eee402df-6e0a-4cac-a025-b981a2ee6329-agent-certs\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207494 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-bin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207521 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-kubelet\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207544 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-bin\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207569 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207596 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzs87\" (UniqueName: \"kubernetes.io/projected/acf17209-903f-4a4e-9baf-908e3ab9afae-kube-api-access-wzs87\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207611 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207651 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-script-lib\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207668 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207708 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgm99\" (UniqueName: \"kubernetes.io/projected/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kube-api-access-zgm99\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207746 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207782 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-serving-cert\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207811 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/6b4895c8-b425-4009-a06b-47f1d51b8784-kube-api-access-8lcbr\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:53.207910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.207837 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a59cf1-454d-4019-9e05-0108e38b8725-serving-cert\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.208483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.208063 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a59cf1-454d-4019-9e05-0108e38b8725-serving-cert\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.208483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.208147 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-serving-cert\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.212936 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.212920 5932 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 20:29:53.246691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.246669 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.247381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247367 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 20:29:53.247527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247500 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 20:29:53.247617 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247547 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 20:29:53.247715 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247640 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.247782 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247720 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.247851 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247830 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 20:29:53.247903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247864 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 20:29:53.247943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247913 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-binary-copy\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.247943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247922 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 20:29:53.247943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.247917 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248132 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248168 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248244 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 20:29:53.248267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248255 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.248356 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248335 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 20:29:53.248406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248397 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248399 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 20:29:53.248454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248435 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.248558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248494 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248516 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 20:29:53.248558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248547 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248645 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.248695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248653 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 20:29:53.248695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248659 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 20:29:53.248827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248704 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 20:29:53.248827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248711 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.248916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248835 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.248916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248869 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248877 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248897 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.248916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248910 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.248916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248916 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248934 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248938 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.248978 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249013 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249002 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249033 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249107 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249108 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 20:29:53.249138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249139 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249212 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249238 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249252 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249298 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-mfn79\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249311 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dfjfq\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249334 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zw8dl\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249351 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249360 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249396 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6c6v9\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249401 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-c56x6\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249429 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249447 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249482 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 20:29:53.249526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249509 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249553 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9m6bz\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249564 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249588 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-t8tnq\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249609 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w66bh\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249645 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jm687\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249675 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249678 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-pdt7f\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249693 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9qtf8\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249690 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bb4f5\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249705 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7m98\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249742 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249766 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9nsh4\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249771 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-245pv\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249784 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249793 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ff7gv\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249787 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9t9mk\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249833 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rrsvs\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249836 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-65mmw\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249852 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249855 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7kkpg\"" Nov 03 20:29:53.250066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.249888 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250168 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250255 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250328 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k7b8b\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250392 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250462 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250501 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 20:29:53.250615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250543 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mx84v\"" Nov 03 20:29:53.250814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.250682 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 20:29:53.251417 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.251404 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 20:29:53.251553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.251540 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 20:29:53.254404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.254384 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.254404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.254397 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-metrics-certs\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:53.254491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.254397 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-metrics-certs\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.254491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.254424 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/536dddb8-cc7a-4b68-a591-7a19d1431afe-iptables-alerter-script\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.254491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.254435 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-default-certificate\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.254491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.254445 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.255607 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255577 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:53.255714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255643 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-config\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.255714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255656 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-cni-binary-copy\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.255714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255668 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7b8419bd-5051-4126-9569-b6344e49c66e-ovn-node-metrics-cert\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.255887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255721 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5e448fa5-814c-427f-854e-02708388f82a-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:53.255887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255760 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5bc681f0-27c6-4cad-8bf6-82835e009976-nginx-conf\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:53.255887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255810 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eee402df-6e0a-4cac-a025-b981a2ee6329-konnectivity-ca\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:53.255887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.255852 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-daemon-config\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.256083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256070 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ab15921-e819-4567-af4a-9d3684c054ae-serviceca\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.256264 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256245 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5e448fa5-814c-427f-854e-02708388f82a-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:53.256264 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256254 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-stats-auth\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.256395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256256 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.256395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256310 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.256395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256373 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c947407f-1cc6-4c01-997e-99e19a3c139c-original-pull-secret\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.256532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256476 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.256532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256523 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-env-overrides\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.256594 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256526 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-service-ca-bundle\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.256947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.256934 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.257021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.257007 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/aac3b6b3-61bd-4ee2-b116-61fb0a786896-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:53.258182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.258162 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7b8419bd-5051-4126-9569-b6344e49c66e-ovnkube-script-lib\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.258182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.258176 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5bc681f0-27c6-4cad-8bf6-82835e009976-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-2b69b\" (UID: \"5bc681f0-27c6-4cad-8bf6-82835e009976\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-2b69b" Nov 03 20:29:53.258309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.258183 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eee402df-6e0a-4cac-a025-b981a2ee6329-agent-certs\") pod \"konnectivity-agent-l5fnw\" (UID: \"eee402df-6e0a-4cac-a025-b981a2ee6329\") " pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:53.258309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.258160 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.258309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.258204 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.258986 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.258970 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlz94\" (UniqueName: \"kubernetes.io/projected/fe95bd0d-a5b5-44db-97ae-d75a6860a71a-kube-api-access-dlz94\") pod \"network-check-target-9x8q2\" (UID: \"fe95bd0d-a5b5-44db-97ae-d75a6860a71a\") " pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:53.293173 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.293145 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdln\" (UniqueName: \"kubernetes.io/projected/2ab15921-e819-4567-af4a-9d3684c054ae-kube-api-access-bwdln\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.293263 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.293148 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlbn\" (UniqueName: \"kubernetes.io/projected/ea25f895-6d8c-4bf1-abd4-6e29c4a772e7-kube-api-access-vtlbn\") pod \"network-metrics-daemon-vfs2r\" (UID: \"ea25f895-6d8c-4bf1-abd4-6e29c4a772e7\") " pod="openshift-multus/network-metrics-daemon-vfs2r" Nov 03 20:29:53.293347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.293328 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9skt\" (UniqueName: \"kubernetes.io/projected/c7a59cf1-454d-4019-9e05-0108e38b8725-kube-api-access-b9skt\") pod \"console-operator-7cd56b8dbc-ljjts\" (UID: \"c7a59cf1-454d-4019-9e05-0108e38b8725\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:53.293444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.293419 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294371 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnfbc\" (UniqueName: \"kubernetes.io/projected/536dddb8-cc7a-4b68-a591-7a19d1431afe-kube-api-access-qnfbc\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294409 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t97p\" (UniqueName: \"kubernetes.io/projected/a7b21223-f0b4-492b-8856-233fa4b1cd58-kube-api-access-9t97p\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294419 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") pod \"image-registry-595485878c-mqdnk\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294444 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d942d\" (UniqueName: \"kubernetes.io/projected/6ab99b87-827c-4aba-a47e-b86a847ca383-kube-api-access-d942d\") pod \"volume-data-source-validator-58c9f5f46d-b74gt\" (UID: \"6ab99b87-827c-4aba-a47e-b86a847ca383\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-b74gt" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294501 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j877w\" (UniqueName: \"kubernetes.io/projected/5e448fa5-814c-427f-854e-02708388f82a-kube-api-access-j877w\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9nws7\" (UID: \"5e448fa5-814c-427f-854e-02708388f82a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294558 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgm99\" (UniqueName: \"kubernetes.io/projected/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kube-api-access-zgm99\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294569 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzs87\" (UniqueName: \"kubernetes.io/projected/acf17209-903f-4a4e-9baf-908e3ab9afae-kube-api-access-wzs87\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294720 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcbr\" (UniqueName: \"kubernetes.io/projected/6b4895c8-b425-4009-a06b-47f1d51b8784-kube-api-access-8lcbr\") pod \"cluster-monitoring-operator-884576c57-627kh\" (UID: \"6b4895c8-b425-4009-a06b-47f1d51b8784\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-627kh" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.294932 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qz8f\" (UniqueName: \"kubernetes.io/projected/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-kube-api-access-9qz8f\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295185 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzg7d\" (UniqueName: \"kubernetes.io/projected/f34740ae-e26e-4188-a405-4da96b57553e-kube-api-access-rzg7d\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295201 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6htdx\" (UniqueName: \"kubernetes.io/projected/6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750-kube-api-access-6htdx\") pod \"kas-connection-checker-685895c7c9-k2tz8\" (UID: \"6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750\") " pod="kube-system/kas-connection-checker-685895c7c9-k2tz8" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295226 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfj4h\" (UniqueName: \"kubernetes.io/projected/f5ca1597-5b32-4231-97cd-e041e4ff6cd4-kube-api-access-nfj4h\") pod \"router-default-76f9745bc9-fxgzq\" (UID: \"f5ca1597-5b32-4231-97cd-e041e4ff6cd4\") " pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.295275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295254 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpnq\" (UniqueName: \"kubernetes.io/projected/7b8419bd-5051-4126-9569-b6344e49c66e-kube-api-access-pbpnq\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.295725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295492 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49dx\" (UniqueName: \"kubernetes.io/projected/98667931-42d6-42d2-b335-57b2d19d7dfc-kube-api-access-p49dx\") pod \"service-ca-operator-544d749c8c-p9rr7\" (UID: \"98667931-42d6-42d2-b335-57b2d19d7dfc\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" Nov 03 20:29:53.295725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295512 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q4l5\" (UniqueName: \"kubernetes.io/projected/a995cca6-542f-406e-a938-aa58715eac9a-kube-api-access-4q4l5\") pod \"kas-connection-checker-685895c7c9-48l5q\" (UID: \"a995cca6-542f-406e-a938-aa58715eac9a\") " pod="kube-system/kas-connection-checker-685895c7c9-48l5q" Nov 03 20:29:53.295725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295615 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rc9\" (UniqueName: \"kubernetes.io/projected/b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4-kube-api-access-z2rc9\") pod \"network-check-source-6f78d59f99-vmfmg\" (UID: \"b60bf0ba-cf9a-4238-bfb2-4b13de61d5b4\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-vmfmg" Nov 03 20:29:53.295725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295658 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxxd9\" (UniqueName: \"kubernetes.io/projected/02bb73c6-e6e2-4d77-8458-20ba459b831a-kube-api-access-qxxd9\") pod \"kas-connection-checker-685895c7c9-8bznz\" (UID: \"02bb73c6-e6e2-4d77-8458-20ba459b831a\") " pod="kube-system/kas-connection-checker-685895c7c9-8bznz" Nov 03 20:29:53.295725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295716 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnlx\" (UniqueName: \"kubernetes.io/projected/ea833ecb-ee75-45fd-b689-2b7c0c9a711c-kube-api-access-swnlx\") pod \"insights-operator-7c4986d449-zd9mv\" (UID: \"ea833ecb-ee75-45fd-b689-2b7c0c9a711c\") " pod="openshift-insights/insights-operator-7c4986d449-zd9mv" Nov 03 20:29:53.295941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.295924 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82xf\" (UniqueName: \"kubernetes.io/projected/aac3b6b3-61bd-4ee2-b116-61fb0a786896-kube-api-access-l82xf\") pod \"cluster-samples-operator-6ddb6c8b94-vqx9b\" (UID: \"aac3b6b3-61bd-4ee2-b116-61fb0a786896\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vqx9b" Nov 03 20:29:53.308461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308441 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-conf-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.308533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308468 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.308533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308483 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.308533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308504 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.308533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308527 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-var-lib-kubelet\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308546 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308565 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-system-cni-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308585 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-lib-modules\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308610 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5l82\" (UniqueName: \"kubernetes.io/projected/3bbdae4d-8429-4f49-8d31-fe00df3d0701-kube-api-access-j5l82\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308613 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-var-lib-kubelet\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308616 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-system-cni-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308650 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308656 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-socket-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308581 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-conf-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308689 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308724 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-metrics-tls\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308735 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-lib-modules\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308741 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-socket-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.308771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308767 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.309207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308794 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-log-socket\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.309207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308799 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.309207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308826 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.309528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309496 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-netd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.309638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309529 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.309638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309539 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.309638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309549 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysconfig\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.308865 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-log-socket\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.309638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309595 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysconfig\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309608 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-systemd\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309654 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-netd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309669 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-multus-certs\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309699 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-modprobe-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309742 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-key\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309779 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-multus-certs\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309781 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f34740ae-e26e-4188-a405-4da96b57553e-hosts-file\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309742 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-systemd\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309837 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f34740ae-e26e-4188-a405-4da96b57553e-hosts-file\") pod \"node-resolver-cwd62\" (UID: \"f34740ae-e26e-4188-a405-4da96b57553e\") " pod="openshift-dns/node-resolver-cwd62" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309865 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bbdae4d-8429-4f49-8d31-fe00df3d0701-cert\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309890 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309923 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-kubernetes\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309937 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-modprobe-d\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.309958 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309951 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-slash\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.309983 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-dbus\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310060 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-key\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310105 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvqhp\" (UniqueName: \"kubernetes.io/projected/b91b91f6-e725-4d7b-b472-9e76650979ab-kube-api-access-kvqhp\") pod \"migrator-66df5d496b-977vh\" (UID: \"b91b91f6-e725-4d7b-b472-9e76650979ab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310139 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-socket-dir-parent\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310163 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-netns\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310173 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-dbus\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310183 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3bbdae4d-8429-4f49-8d31-fe00df3d0701-cert\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310241 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-kubernetes\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310261 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-socket-dir-parent\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310273 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-system-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310345 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-system-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310373 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-netns\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310384 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-systemd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310424 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-systemd\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310436 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310467 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-slash\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310469 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.310510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310516 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/536dddb8-cc7a-4b68-a591-7a19d1431afe-host-slash\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310548 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-etc-kubernetes\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310599 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310648 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310678 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-sys-fs\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310721 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-systemd-units\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310752 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cnibin\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310782 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310798 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-config-volume\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310782 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-etc-kubernetes\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310844 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/536dddb8-cc7a-4b68-a591-7a19d1431afe-host-slash\") pod \"iptables-alerter-qkfwr\" (UID: \"536dddb8-cc7a-4b68-a591-7a19d1431afe\") " pod="openshift-network-operator/iptables-alerter-qkfwr" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310866 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-hostroot\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310898 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-sys-fs\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310903 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-run\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310944 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-sys\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310962 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-hostroot\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311001 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-sys\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311011 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-run\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.310898 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-cnibin\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311061 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-systemd-units\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.311334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311294 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311426 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-config-volume\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311480 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmm8\" (UniqueName: \"kubernetes.io/projected/3086db43-8d5a-4196-ab9e-8c7f7832c232-kube-api-access-jwmm8\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311524 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311531 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-multus\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311573 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311588 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-multus\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311603 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ab15921-e819-4567-af4a-9d3684c054ae-host\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311646 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52568b14-d428-42c1-af42-2e15763fc1b3-data-volume\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311680 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv629\" (UniqueName: \"kubernetes.io/projected/ff18aebf-b9af-4296-aaad-5d24fd9304ea-kube-api-access-kv629\") pod \"migrator-66df5d496b-rn27d\" (UID: \"ff18aebf-b9af-4296-aaad-5d24fd9304ea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311711 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-kubelet\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311741 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-node-log\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311772 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311799 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-k8s-cni-cncf-io\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311827 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-netns\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311857 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311918 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.311983 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312024 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ab15921-e819-4567-af4a-9d3684c054ae-host\") pod \"node-ca-2j4jr\" (UID: \"2ab15921-e819-4567-af4a-9d3684c054ae\") " pod="openshift-image-registry/node-ca-2j4jr" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312091 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52568b14-d428-42c1-af42-2e15763fc1b3-data-volume\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312147 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-ovn\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312192 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-kubelet-config\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.312215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312227 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkmm\" (UniqueName: \"kubernetes.io/projected/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-kube-api-access-6pkmm\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312274 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312309 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52568b14-d428-42c1-af42-2e15763fc1b3-crio-socket\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312337 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-kubelet\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312352 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-bin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312416 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-cni-bin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312461 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-node-log\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312525 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-ovn\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312565 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312593 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c947407f-1cc6-4c01-997e-99e19a3c139c-kubelet-config\") pod \"global-pull-secret-syncer-5khcq\" (UID: \"c947407f-1cc6-4c01-997e-99e19a3c139c\") " pod="kube-system/global-pull-secret-syncer-5khcq" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312634 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-k8s-cni-cncf-io\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312675 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-run-netns\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312717 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-multus-cni-dir\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312741 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-kubelet\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312783 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312815 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-bin\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312826 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-host-var-lib-kubelet\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312843 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312960 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.312997 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313026 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-tmp-dir\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313059 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpv4\" (UniqueName: \"kubernetes.io/projected/52568b14-d428-42c1-af42-2e15763fc1b3-kube-api-access-smpv4\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313062 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52568b14-d428-42c1-af42-2e15763fc1b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.313136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313148 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/311c1fbb-32b5-4d7d-934b-28b68c8d48cb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8rvtd\" (UID: \"311c1fbb-32b5-4d7d-934b-28b68c8d48cb\") " pod="openshift-multus/multus-additional-cni-plugins-8rvtd" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313207 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-cni-bin\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313293 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-cnibin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313302 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-tmp-dir\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313330 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-registration-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313361 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-device-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313366 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313387 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-host\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313420 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313435 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7b21223-f0b4-492b-8856-233fa4b1cd58-cnibin\") pod \"multus-qxfrx\" (UID: \"a7b21223-f0b4-492b-8856-233fa4b1cd58\") " pod="openshift-multus/multus-qxfrx" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313455 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313482 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-device-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313485 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-var-lib-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313582 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-run-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313648 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-var-lib-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313689 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-host\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313743 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-etc-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313775 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313834 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-cabundle\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313881 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-conf\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.313906 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.314121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314092 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.315013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314199 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f0d106f-4b85-48cf-8c77-6b4966cfd1f3-registration-dir\") pod \"aws-ebs-csi-driver-node-gwqm2\" (UID: \"7f0d106f-4b85-48cf-8c77-6b4966cfd1f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gwqm2" Nov 03 20:29:53.315013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314275 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-etc-openvswitch\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.315013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314326 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b8419bd-5051-4126-9569-b6344e49c66e-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6gnn\" (UID: \"7b8419bd-5051-4126-9569-b6344e49c66e\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:53.315013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314326 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52568b14-d428-42c1-af42-2e15763fc1b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.315013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314459 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/acf17209-903f-4a4e-9baf-908e3ab9afae-etc-sysctl-conf\") pod \"tuned-2fhq4\" (UID: \"acf17209-903f-4a4e-9baf-908e3ab9afae\") " pod="openshift-cluster-node-tuning-operator/tuned-2fhq4" Nov 03 20:29:53.315013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.314546 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3086db43-8d5a-4196-ab9e-8c7f7832c232-signing-cabundle\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:53.363466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.363444 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5l82\" (UniqueName: \"kubernetes.io/projected/3bbdae4d-8429-4f49-8d31-fe00df3d0701-kube-api-access-j5l82\") pod \"ingress-canary-bw59n\" (UID: \"3bbdae4d-8429-4f49-8d31-fe00df3d0701\") " pod="openshift-ingress-canary/ingress-canary-bw59n" Nov 03 20:29:53.364969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.364952 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.365022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.364967 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvqhp\" (UniqueName: \"kubernetes.io/projected/b91b91f6-e725-4d7b-b472-9e76650979ab-kube-api-access-kvqhp\") pod \"migrator-66df5d496b-977vh\" (UID: \"b91b91f6-e725-4d7b-b472-9e76650979ab\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-977vh" Nov 03 20:29:53.366082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.366062 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") pod \"image-registry-7b5c54f4d6-pwkct\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:53.366208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.366191 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv629\" (UniqueName: \"kubernetes.io/projected/ff18aebf-b9af-4296-aaad-5d24fd9304ea-kube-api-access-kv629\") pod \"migrator-66df5d496b-rn27d\" (UID: \"ff18aebf-b9af-4296-aaad-5d24fd9304ea\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rn27d" Nov 03 20:29:53.366359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.366343 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmm8\" (UniqueName: \"kubernetes.io/projected/3086db43-8d5a-4196-ab9e-8c7f7832c232-kube-api-access-jwmm8\") pod \"service-ca-5fffc97bdf-x5fbx\" (UID: \"3086db43-8d5a-4196-ab9e-8c7f7832c232\") " pod="openshift-service-ca/service-ca-5fffc97bdf-x5fbx" Nov 03 20:29:53.366850 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.366835 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkmm\" (UniqueName: \"kubernetes.io/projected/e44f243f-ff0c-4e9a-9a91-b0079c2667b6-kube-api-access-6pkmm\") pod \"dns-default-sd2tb\" (UID: \"e44f243f-ff0c-4e9a-9a91-b0079c2667b6\") " pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:53.367334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.367319 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpv4\" (UniqueName: \"kubernetes.io/projected/52568b14-d428-42c1-af42-2e15763fc1b3-kube-api-access-smpv4\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.369406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.369390 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") pod \"periodic-gathering-lhwt8-bjdh2\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:29:53.402869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.402849 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:53.414805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.414789 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52568b14-d428-42c1-af42-2e15763fc1b3-crio-socket\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.414854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.414833 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52568b14-d428-42c1-af42-2e15763fc1b3-crio-socket\") pod \"insights-runtime-extractor-dpsrk\" (UID: \"52568b14-d428-42c1-af42-2e15763fc1b3\") " pod="openshift-insights/insights-runtime-extractor-dpsrk" Nov 03 20:29:53.606820 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:53.606745 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:54.404081 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:54.404054 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:54.949736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:54.949711 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sd2tb" Nov 03 20:29:55.142220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:55.142186 5932 generic.go:356] "Generic (PLEG): container finished" podID="5e448fa5-814c-427f-854e-02708388f82a" containerID="9f28042b548d8cf83c78c57a070d3353b6252c7cafc0d076fd4a886c289c907f" exitCode=0 Nov 03 20:29:55.142372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:55.142252 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" event={"ID":"5e448fa5-814c-427f-854e-02708388f82a","Type":"ContainerDied","Data":"9f28042b548d8cf83c78c57a070d3353b6252c7cafc0d076fd4a886c289c907f"} Nov 03 20:29:55.142812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:55.142786 5932 scope.go:122] "RemoveContainer" containerID="9f28042b548d8cf83c78c57a070d3353b6252c7cafc0d076fd4a886c289c907f" Nov 03 20:29:55.143822 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:55.143804 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76f9745bc9-fxgzq" Nov 03 20:29:55.629088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:55.629060 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-ljjts" Nov 03 20:29:55.896512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:55.896432 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9x8q2" Nov 03 20:29:56.108835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:56.108810 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:29:56.127894 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:56.127873 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l5fnw" Nov 03 20:29:56.147204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:56.147145 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9nws7" event={"ID":"5e448fa5-814c-427f-854e-02708388f82a","Type":"ContainerStarted","Data":"234a6aa754ec4b73423d38be98e94452c9b8dc4eb19c3d7376b4d99ecdd41daa"} Nov 03 20:29:57.839838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:57.839774 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:29:58.158806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:58.158776 5932 generic.go:356] "Generic (PLEG): container finished" podID="98667931-42d6-42d2-b335-57b2d19d7dfc" containerID="a6215f8eba2cccd47d7cd2bf203504506ac431492422a68707a7b4ac1f2649b5" exitCode=0 Nov 03 20:29:58.158948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:58.158848 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" event={"ID":"98667931-42d6-42d2-b335-57b2d19d7dfc","Type":"ContainerDied","Data":"a6215f8eba2cccd47d7cd2bf203504506ac431492422a68707a7b4ac1f2649b5"} Nov 03 20:29:58.159152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:58.159135 5932 scope.go:122] "RemoveContainer" containerID="a6215f8eba2cccd47d7cd2bf203504506ac431492422a68707a7b4ac1f2649b5" Nov 03 20:29:59.166070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:59.166038 5932 generic.go:356] "Generic (PLEG): container finished" podID="ea833ecb-ee75-45fd-b689-2b7c0c9a711c" containerID="532206674ebd8f316be3a2000f31ee2e745eb7fd7d8ad83f2b546bdf9109d3d4" exitCode=0 Nov 03 20:29:59.166483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:59.166115 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" event={"ID":"ea833ecb-ee75-45fd-b689-2b7c0c9a711c","Type":"ContainerDied","Data":"532206674ebd8f316be3a2000f31ee2e745eb7fd7d8ad83f2b546bdf9109d3d4"} Nov 03 20:29:59.166483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:59.166470 5932 scope.go:122] "RemoveContainer" containerID="532206674ebd8f316be3a2000f31ee2e745eb7fd7d8ad83f2b546bdf9109d3d4" Nov 03 20:29:59.168143 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:29:59.168123 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-p9rr7" event={"ID":"98667931-42d6-42d2-b335-57b2d19d7dfc","Type":"ContainerStarted","Data":"b17956da0aae987434de3f91cef790d0ede7534128cf6694ca35b752f7dffd34"} Nov 03 20:30:00.174477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.174442 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-zd9mv" event={"ID":"ea833ecb-ee75-45fd-b689-2b7c0c9a711c","Type":"ContainerStarted","Data":"b23e91aba2ad884c0c1bcef0b618b23078c9ab788db5f0186d7b3ce2e9e87115"} Nov 03 20:30:00.660058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.660022 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sqx4f"] Nov 03 20:30:00.662478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.662455 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.758429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.758406 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 20:30:00.758537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.758456 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 20:30:00.767898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.767875 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.767912 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-sys\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.767934 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-wtmp\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.768004 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-tls\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.768072 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9496v\" (UniqueName: \"kubernetes.io/projected/aefd9a58-f889-4651-9b0b-f7bfe254355f-kube-api-access-9496v\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.768104 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-textfile\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.768171 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aefd9a58-f889-4651-9b0b-f7bfe254355f-metrics-client-ca\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.768200 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-root\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.768235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.768217 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.797475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.797450 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 20:30:00.797713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.797695 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 20:30:00.798058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.798043 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4nsd9\"" Nov 03 20:30:00.868726 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868702 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aefd9a58-f889-4651-9b0b-f7bfe254355f-metrics-client-ca\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.868812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868738 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-root\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.868812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868759 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.868812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868799 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.868907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868828 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-root\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.868907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868836 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-sys\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868929 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-wtmp\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868870 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-sys\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.868960 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-tls\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869043 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9496v\" (UniqueName: \"kubernetes.io/projected/aefd9a58-f889-4651-9b0b-f7bfe254355f-kube-api-access-9496v\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869077 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-textfile\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869122 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-wtmp\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869159 5932 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 20:30:00.869412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869394 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aefd9a58-f889-4651-9b0b-f7bfe254355f-metrics-client-ca\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869427 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.869469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.869433 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-textfile\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.871238 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.871219 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-tls\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.871314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.871283 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aefd9a58-f889-4651-9b0b-f7bfe254355f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.926498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.926448 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9496v\" (UniqueName: \"kubernetes.io/projected/aefd9a58-f889-4651-9b0b-f7bfe254355f-kube-api-access-9496v\") pod \"node-exporter-sqx4f\" (UID: \"aefd9a58-f889-4651-9b0b-f7bfe254355f\") " pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.975600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.975566 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqx4f" Nov 03 20:30:00.986405 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:30:00.986376 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefd9a58_f889_4651_9b0b_f7bfe254355f.slice/crio-28c41bdba90b39cfa87b30a36a61444ff967414c6e89e38563488dcc851cc2fc WatchSource:0}: Error finding container 28c41bdba90b39cfa87b30a36a61444ff967414c6e89e38563488dcc851cc2fc: Status 404 returned error can't find the container with id 28c41bdba90b39cfa87b30a36a61444ff967414c6e89e38563488dcc851cc2fc Nov 03 20:30:00.988287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:00.988272 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:30:01.180765 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:01.180682 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqx4f" event={"ID":"aefd9a58-f889-4651-9b0b-f7bfe254355f","Type":"ContainerStarted","Data":"28c41bdba90b39cfa87b30a36a61444ff967414c6e89e38563488dcc851cc2fc"} Nov 03 20:30:01.430403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:01.430377 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:30:01.430694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:01.430676 5932 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:30:03.198092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:03.198049 5932 generic.go:356] "Generic (PLEG): container finished" podID="aefd9a58-f889-4651-9b0b-f7bfe254355f" containerID="441a42e2dbe8b8509dc73d244062a3bd2166e132bc6256cfc32d20c77717c114" exitCode=0 Nov 03 20:30:03.198530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:03.198124 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqx4f" event={"ID":"aefd9a58-f889-4651-9b0b-f7bfe254355f","Type":"ContainerDied","Data":"441a42e2dbe8b8509dc73d244062a3bd2166e132bc6256cfc32d20c77717c114"} Nov 03 20:30:04.204489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:04.204458 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqx4f" event={"ID":"aefd9a58-f889-4651-9b0b-f7bfe254355f","Type":"ContainerStarted","Data":"c1a7fd7fba3e502b9025351d0fa697face4e43b53ef2b8d76a97ed9fbf20928d"} Nov 03 20:30:04.204905 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:04.204509 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqx4f" event={"ID":"aefd9a58-f889-4651-9b0b-f7bfe254355f","Type":"ContainerStarted","Data":"6b1518b276163d6f1a9b020a8735d9c64488a4aaedc43ca0d1a9a6176ce0b1be"} Nov 03 20:30:04.369086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:04.369037 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sqx4f" podStartSLOduration=2.67707011 podStartE2EDuration="4.369022864s" totalImagesPullingTime="1.623699327s" totalInitContainerRuntime="68.253427ms" isStatefulPod=true podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.36874893 +0000 UTC m=+12.433990403" watchObservedRunningTime="2026-11-03 20:30:04.369022864 +0000 UTC m=+12.434264333" Nov 03 20:30:06.475008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:06.474970 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b5c54f4d6-pwkct"] Nov 03 20:30:07.274212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:07.274165 5932 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l5fnw_eee402df-6e0a-4cac-a025-b981a2ee6329/konnectivity-agent/0.log" limit=1024 Nov 03 20:30:14.105460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.105422 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:30:14.105916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.105616 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" podUID="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" containerName="insights-gathering" containerID="cri-o://d21b2270103421f77f86dd2df622b214d341e6180f52113e366fd567a8685649" gracePeriod=30 Nov 03 20:30:14.150172 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:30:14.150138 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:30:14.247812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.247782 5932 generic.go:356] "Generic (PLEG): container finished" podID="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" containerID="d21b2270103421f77f86dd2df622b214d341e6180f52113e366fd567a8685649" exitCode=2 Nov 03 20:30:14.247943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.247861 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" event={"ID":"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a","Type":"ContainerDied","Data":"d21b2270103421f77f86dd2df622b214d341e6180f52113e366fd567a8685649"} Nov 03 20:30:14.362039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.361982 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:30:14.492590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.492558 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") pod \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " Nov 03 20:30:14.492590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.492594 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") pod \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " Nov 03 20:30:14.492812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.492685 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") pod \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " Nov 03 20:30:14.492886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.492858 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") pod \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\" (UID: \"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a\") " Nov 03 20:30:14.493015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.492991 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle" pod "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" (UID: "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:14.493161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.493139 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle" pod "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" (UID: "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:14.493222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.493153 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path" pod "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" (UID: "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:14.493222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.493171 5932 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-service-ca-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:14.495095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.495073 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8" pod "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" (UID: "45ab6c3f-8d7c-440d-a7d8-4917dc10f26a"). InnerVolumeSpecName "kube-api-access-ffww8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:14.593789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.593757 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffww8\" (UniqueName: \"kubernetes.io/projected/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-kube-api-access-ffww8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:14.593789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.593783 5932 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-archives-path\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:14.593789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:14.593794 5932 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a-trusted-ca-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:15.253322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:15.253242 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" event={"ID":"45ab6c3f-8d7c-440d-a7d8-4917dc10f26a","Type":"ContainerDied","Data":"ca093a850baa38255663e858e702bab90a324de6ccda634101c13e63d60b831e"} Nov 03 20:30:15.253322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:15.253298 5932 scope.go:122] "RemoveContainer" containerID="d21b2270103421f77f86dd2df622b214d341e6180f52113e366fd567a8685649" Nov 03 20:30:15.253794 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:15.253251 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-lhwt8-bjdh2" Nov 03 20:30:15.406266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:15.406214 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:30:15.451051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:15.451020 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-lhwt8-bjdh2"] Nov 03 20:30:16.049319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:16.049286 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" path="/var/lib/kubelet/pods/45ab6c3f-8d7c-440d-a7d8-4917dc10f26a/volumes" Nov 03 20:30:17.032633 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:17.032591 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6gnn" Nov 03 20:30:18.131857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.131816 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-595485878c-mqdnk" podUID="4382cc48-e16d-4e80-85af-ff5f5d715c7a" containerName="registry" containerID="cri-o://e050b87363d833b7b8b478adfc65719fa3127c8196a92b4b5e9e8a6da93ec420" gracePeriod=30 Nov 03 20:30:18.269551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.269517 5932 generic.go:356] "Generic (PLEG): container finished" podID="4382cc48-e16d-4e80-85af-ff5f5d715c7a" containerID="e050b87363d833b7b8b478adfc65719fa3127c8196a92b4b5e9e8a6da93ec420" exitCode=0 Nov 03 20:30:18.269697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.269587 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595485878c-mqdnk" event={"ID":"4382cc48-e16d-4e80-85af-ff5f5d715c7a","Type":"ContainerDied","Data":"e050b87363d833b7b8b478adfc65719fa3127c8196a92b4b5e9e8a6da93ec420"} Nov 03 20:30:18.389198 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.389142 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:30:18.528789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.528759 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.528951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.528804 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.528951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.528846 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.529039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.529012 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.529083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.529066 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.529131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.529107 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.529187 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.529140 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.529242 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.529217 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") pod \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\" (UID: \"4382cc48-e16d-4e80-85af-ff5f5d715c7a\") " Nov 03 20:30:18.529551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.529466 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:18.530174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.530140 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:18.531632 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.531594 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.531736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.531611 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:18.531736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.531699 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.531853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.531838 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:18.531921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.531897 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "kube-api-access-9f59t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.536559 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.536537 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted" pod "4382cc48-e16d-4e80-85af-ff5f5d715c7a" (UID: "4382cc48-e16d-4e80-85af-ff5f5d715c7a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:18.630001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.629973 5932 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-trusted-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.629996 5932 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4382cc48-e16d-4e80-85af-ff5f5d715c7a-ca-trust-extracted\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.630007 5932 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-image-registry-private-configuration\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.630017 5932 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-bound-sa-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.630026 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f59t\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-kube-api-access-9f59t\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.630037 5932 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.630045 5932 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4382cc48-e16d-4e80-85af-ff5f5d715c7a-registry-certificates\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.630214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:18.630054 5932 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4382cc48-e16d-4e80-85af-ff5f5d715c7a-installation-pull-secrets\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.275638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:19.275588 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-595485878c-mqdnk" Nov 03 20:30:19.275638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:19.275605 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-595485878c-mqdnk" event={"ID":"4382cc48-e16d-4e80-85af-ff5f5d715c7a","Type":"ContainerDied","Data":"c5429167018357a260f55a8147d5793e7cf133eb12f68f79861893412d125914"} Nov 03 20:30:19.276095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:19.275677 5932 scope.go:122] "RemoveContainer" containerID="e050b87363d833b7b8b478adfc65719fa3127c8196a92b4b5e9e8a6da93ec420" Nov 03 20:30:19.478292 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:19.478260 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-595485878c-mqdnk"] Nov 03 20:30:19.524732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:19.524699 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-595485878c-mqdnk"] Nov 03 20:30:20.048549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:20.048516 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4382cc48-e16d-4e80-85af-ff5f5d715c7a" path="/var/lib/kubelet/pods/4382cc48-e16d-4e80-85af-ff5f5d715c7a/volumes" Nov 03 20:30:31.494770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.494709 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" podUID="abc63a7c-b9ed-4269-90aa-49a71a94d932" containerName="registry" containerID="cri-o://c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af" gracePeriod=30 Nov 03 20:30:31.753072 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.753019 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:30:31.937759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937724 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.937946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937776 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.937946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937807 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.937946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937854 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.937946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937875 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.937946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937910 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.937946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937946 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.938267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.937991 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") pod \"abc63a7c-b9ed-4269-90aa-49a71a94d932\" (UID: \"abc63a7c-b9ed-4269-90aa-49a71a94d932\") " Nov 03 20:30:31.938267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.938223 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:31.938549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.938521 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:31.940832 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.940806 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:31.940938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.940901 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:31.940938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.940915 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:31.941068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.941023 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "kube-api-access-qdwdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:31.941125 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.941073 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:31.945419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:31.945230 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted" pod "abc63a7c-b9ed-4269-90aa-49a71a94d932" (UID: "abc63a7c-b9ed-4269-90aa-49a71a94d932"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:32.039011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.038935 5932 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-installation-pull-secrets\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.038964 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwdc\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-kube-api-access-qdwdc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.038979 5932 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-bound-sa-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.038991 5932 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-certificates\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.039004 5932 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/abc63a7c-b9ed-4269-90aa-49a71a94d932-ca-trust-extracted\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.039017 5932 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/abc63a7c-b9ed-4269-90aa-49a71a94d932-trusted-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.039030 5932 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/abc63a7c-b9ed-4269-90aa-49a71a94d932-image-registry-private-configuration\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.039250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.039043 5932 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/abc63a7c-b9ed-4269-90aa-49a71a94d932-registry-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:32.326678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.326564 5932 generic.go:356] "Generic (PLEG): container finished" podID="abc63a7c-b9ed-4269-90aa-49a71a94d932" containerID="c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af" exitCode=0 Nov 03 20:30:32.326678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.326641 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" Nov 03 20:30:32.326873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.326654 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" event={"ID":"abc63a7c-b9ed-4269-90aa-49a71a94d932","Type":"ContainerDied","Data":"c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af"} Nov 03 20:30:32.326873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.326775 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5c54f4d6-pwkct" event={"ID":"abc63a7c-b9ed-4269-90aa-49a71a94d932","Type":"ContainerDied","Data":"7f5f4c854dbe334578b20b6de376ade7a723167dd5879d296b348ca594cad71b"} Nov 03 20:30:32.326873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.326799 5932 scope.go:122] "RemoveContainer" containerID="c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af" Nov 03 20:30:32.334883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.334863 5932 scope.go:122] "RemoveContainer" containerID="c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af" Nov 03 20:30:32.335146 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:30:32.335125 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af\": container with ID starting with c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af not found: ID does not exist" containerID="c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af" Nov 03 20:30:32.335219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.335154 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af"} err="failed to get container status \"c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af\": rpc error: code = NotFound desc = could not find container \"c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af\": container with ID starting with c9e4329195fcb535c2fbcaed33aeb075e8a10c21e0aa19824e6feb556bd1e6af not found: ID does not exist" Nov 03 20:30:32.479016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.478985 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b5c54f4d6-pwkct"] Nov 03 20:30:32.525048 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:32.525020 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b5c54f4d6-pwkct"] Nov 03 20:30:34.048636 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:30:34.048587 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc63a7c-b9ed-4269-90aa-49a71a94d932" path="/var/lib/kubelet/pods/abc63a7c-b9ed-4269-90aa-49a71a94d932/volumes" Nov 03 20:30:44.149087 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:30:44.149050 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:31:14.150025 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:31:14.149987 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:31:44.148714 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:31:44.148613 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:32:14.150244 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:32:14.150208 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:32:44.148417 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:32:44.148374 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:33:14.149363 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:33:14.149288 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:33:44.148238 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:33:44.148205 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:34:14.149304 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:34:14.149268 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:34:44.149272 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:34:44.149194 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:34:44.499601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499526 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5"] Nov 03 20:34:44.499864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499841 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4382cc48-e16d-4e80-85af-ff5f5d715c7a" containerName="registry" Nov 03 20:34:44.499864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499859 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4382cc48-e16d-4e80-85af-ff5f5d715c7a" containerName="registry" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499895 5932 cpu_manager.go:414] "containerMap: removing container" podUID="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" containerName="insights-gathering" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499900 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" containerName="insights-gathering" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499914 5932 cpu_manager.go:414] "containerMap: removing container" podUID="abc63a7c-b9ed-4269-90aa-49a71a94d932" containerName="registry" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499919 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc63a7c-b9ed-4269-90aa-49a71a94d932" containerName="registry" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499979 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ab6c3f-8d7c-440d-a7d8-4917dc10f26a" containerName="insights-gathering" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499988 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc63a7c-b9ed-4269-90aa-49a71a94d932" containerName="registry" Nov 03 20:34:44.500010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.499994 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4382cc48-e16d-4e80-85af-ff5f5d715c7a" containerName="registry" Nov 03 20:34:44.503080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.503062 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.578904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.578873 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.579038 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.578930 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhm4\" (UniqueName: \"kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.579038 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.579030 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.597696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.597668 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-2gmr6\"" Nov 03 20:34:44.597802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.597738 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:34:44.597802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.597789 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:34:44.598457 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.598440 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5"] Nov 03 20:34:44.601581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.601565 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv"] Nov 03 20:34:44.604882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.604866 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.656755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.656736 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-pxsq5\"" Nov 03 20:34:44.661319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.661297 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv"] Nov 03 20:34:44.679939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.679918 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.680031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.679957 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.680031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.680026 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhm4\" (UniqueName: \"kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.680119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.680063 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.680119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.680099 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ldgb\" (UniqueName: \"kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.680205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.680136 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.680354 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.680336 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.680411 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.680390 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.704952 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.704923 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr"] Nov 03 20:34:44.708319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.708301 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.737136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.737109 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhm4\" (UniqueName: \"kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.758249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.758196 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-kzbcz\"" Nov 03 20:34:44.762762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.762739 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr"] Nov 03 20:34:44.780540 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780519 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.780660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780549 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.780660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780591 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ldgb\" (UniqueName: \"kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.780750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780672 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmhbm\" (UniqueName: \"kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.780750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780710 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.780824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780764 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.781005 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.780990 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.781046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.781016 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.804513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.804484 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx"] Nov 03 20:34:44.807943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.807924 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.817467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.817448 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:34:44.837448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.837423 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ldgb\" (UniqueName: \"kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.856359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.856320 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h9tct\"" Nov 03 20:34:44.862339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.862303 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx"] Nov 03 20:34:44.881981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.881494 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27q2\" (UniqueName: \"kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.881981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.881531 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.881981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.881562 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmhbm\" (UniqueName: \"kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.881981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.881599 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.881981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.881632 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.881981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.881663 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.882724 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.882675 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.882724 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.882689 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.919355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.919326 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:34:44.938698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.938674 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmhbm\" (UniqueName: \"kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:44.982327 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.982306 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.982468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.982343 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27q2\" (UniqueName: \"kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.982468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.982368 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.982707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.982688 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:44.982771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:44.982752 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:45.023145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.023088 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:34:45.039998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.039967 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27q2\" (UniqueName: \"kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:45.123799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.123768 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:34:45.188574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.188549 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5"] Nov 03 20:34:45.198402 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:45.198375 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35f0226_f88d_44e2_8ca5_beb89ba661b6.slice/crio-859077a428b745d583c68445e2d59d49347ccd56a4f0d918001e777ad1bbee5d WatchSource:0}: Error finding container 859077a428b745d583c68445e2d59d49347ccd56a4f0d918001e777ad1bbee5d: Status 404 returned error can't find the container with id 859077a428b745d583c68445e2d59d49347ccd56a4f0d918001e777ad1bbee5d Nov 03 20:34:45.222561 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.222522 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" event={"ID":"b35f0226-f88d-44e2-8ca5-beb89ba661b6","Type":"ContainerStarted","Data":"859077a428b745d583c68445e2d59d49347ccd56a4f0d918001e777ad1bbee5d"} Nov 03 20:34:45.284814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.284746 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt"] Nov 03 20:34:45.289428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.289410 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.291090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.291063 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv"] Nov 03 20:34:45.338338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.338319 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-85lgh\"" Nov 03 20:34:45.342685 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.342664 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt"] Nov 03 20:34:45.385538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.385513 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.385732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.385545 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.385732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.385566 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnl7s\" (UniqueName: \"kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.390942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.390920 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr"] Nov 03 20:34:45.398533 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:45.398512 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c552d8_2112_4dd6_ab98_3da570acf7c8.slice/crio-ae931cf80f8860f0cbd9d53d67456f3ab19e85f0abeedbbeaabb9cb4a0e9be5b WatchSource:0}: Error finding container ae931cf80f8860f0cbd9d53d67456f3ab19e85f0abeedbbeaabb9cb4a0e9be5b: Status 404 returned error can't find the container with id ae931cf80f8860f0cbd9d53d67456f3ab19e85f0abeedbbeaabb9cb4a0e9be5b Nov 03 20:34:45.486053 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.486028 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.486201 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.486058 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.486201 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.486084 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnl7s\" (UniqueName: \"kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.486390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.486372 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.486467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.486446 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.489478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.489456 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx"] Nov 03 20:34:45.496939 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:45.496917 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ef0414_a6cd_4424_8d1b_65fe82502b89.slice/crio-f9b9c4effbae6058f510ad9216148df287987a09a91a0a9c21698315f1290e3d WatchSource:0}: Error finding container f9b9c4effbae6058f510ad9216148df287987a09a91a0a9c21698315f1290e3d: Status 404 returned error can't find the container with id f9b9c4effbae6058f510ad9216148df287987a09a91a0a9c21698315f1290e3d Nov 03 20:34:45.543371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.543317 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnl7s\" (UniqueName: \"kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.610225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.610194 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:34:45.977941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:45.977913 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt"] Nov 03 20:34:45.985550 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:45.985520 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f773a6e_5bab_4b86_9eb5_8e8d0d9ed467.slice/crio-a88502dd3be9cbfc7d6109adbce0832caea6dfce0428d1d555ccdee3f1694311 WatchSource:0}: Error finding container a88502dd3be9cbfc7d6109adbce0832caea6dfce0428d1d555ccdee3f1694311: Status 404 returned error can't find the container with id a88502dd3be9cbfc7d6109adbce0832caea6dfce0428d1d555ccdee3f1694311 Nov 03 20:34:46.228233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:46.228150 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" event={"ID":"20ef0414-a6cd-4424-8d1b-65fe82502b89","Type":"ContainerStarted","Data":"f9b9c4effbae6058f510ad9216148df287987a09a91a0a9c21698315f1290e3d"} Nov 03 20:34:46.229391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:46.229370 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" event={"ID":"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467","Type":"ContainerStarted","Data":"a88502dd3be9cbfc7d6109adbce0832caea6dfce0428d1d555ccdee3f1694311"} Nov 03 20:34:46.230427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:46.230402 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" event={"ID":"51c552d8-2112-4dd6-ab98-3da570acf7c8","Type":"ContainerStarted","Data":"ae931cf80f8860f0cbd9d53d67456f3ab19e85f0abeedbbeaabb9cb4a0e9be5b"} Nov 03 20:34:46.231442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:46.231423 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" event={"ID":"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2","Type":"ContainerStarted","Data":"f50ab674dac85a3cddaaea51285ceef22998bf7fcbaffb3e25732ce3fb188597"} Nov 03 20:34:47.320111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.317790 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj"] Nov 03 20:34:47.323333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.323307 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.404740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.404710 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.404976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.404765 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhd9p\" (UniqueName: \"kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.404976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.404810 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.422664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.421702 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-jlrpf\"" Nov 03 20:34:47.422664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.422655 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj"] Nov 03 20:34:47.507517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.506160 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhd9p\" (UniqueName: \"kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.507517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.506228 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.507517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.506322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.507517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.506839 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.507517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.507434 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.567246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.567213 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhd9p\" (UniqueName: \"kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:47.645696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:47.645351 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:34:48.053668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:48.053561 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj"] Nov 03 20:34:48.086298 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:48.086254 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0b12fd_4cdc_421a_8bd5_22228937eab3.slice/crio-4c744f4a8e4e7a7cdf9104f1f8bbfe957d6aae2ed27ed33dd21b37e4935027b6 WatchSource:0}: Error finding container 4c744f4a8e4e7a7cdf9104f1f8bbfe957d6aae2ed27ed33dd21b37e4935027b6: Status 404 returned error can't find the container with id 4c744f4a8e4e7a7cdf9104f1f8bbfe957d6aae2ed27ed33dd21b37e4935027b6 Nov 03 20:34:48.250771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:48.250734 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" event={"ID":"fa0b12fd-4cdc-421a-8bd5-22228937eab3","Type":"ContainerStarted","Data":"4c744f4a8e4e7a7cdf9104f1f8bbfe957d6aae2ed27ed33dd21b37e4935027b6"} Nov 03 20:34:49.722609 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.721826 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s"] Nov 03 20:34:49.727711 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.727054 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.826260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.825287 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s"] Nov 03 20:34:49.826260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.825634 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-c55wx\"" Nov 03 20:34:49.830045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.829762 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92q4w\" (UniqueName: \"kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.830045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.829830 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.830045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.829898 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.930420 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.930376 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92q4w\" (UniqueName: \"kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.930616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.930451 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.930616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.930511 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.930961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.930939 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.931558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.931533 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:49.996279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:49.996212 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92q4w\" (UniqueName: \"kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:50.056810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:50.056746 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:34:50.461479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:50.461443 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s"] Nov 03 20:34:50.475191 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:50.475134 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c5a0c0_abe1_470f_85e5_67ec9ad61762.slice/crio-fb4b8582ccef73b4fb2914732d527d7ebe009a333e0dd1081db3c1214be80479 WatchSource:0}: Error finding container fb4b8582ccef73b4fb2914732d527d7ebe009a333e0dd1081db3c1214be80479: Status 404 returned error can't find the container with id fb4b8582ccef73b4fb2914732d527d7ebe009a333e0dd1081db3c1214be80479 Nov 03 20:34:51.279497 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:51.279460 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" event={"ID":"92c5a0c0-abe1-470f-85e5-67ec9ad61762","Type":"ContainerStarted","Data":"fb4b8582ccef73b4fb2914732d527d7ebe009a333e0dd1081db3c1214be80479"} Nov 03 20:34:52.102352 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.102320 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:34:52.103029 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.103003 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:34:52.515164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.515128 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7"] Nov 03 20:34:52.519981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.519116 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.560714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560678 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.560862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560730 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.560862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560792 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.560862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560830 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.560862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560858 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.561089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560888 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.561089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560928 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.561089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560954 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tph6p\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.561089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.560992 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.569165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.569138 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:34:52.591568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.591485 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\"" Nov 03 20:34:52.617080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.617052 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7"] Nov 03 20:34:52.662149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662114 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662162 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tph6p\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662205 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662266 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662295 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662612 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662347 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662612 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662364 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.662612 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662383 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.665709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662781 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.665709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.662961 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.665709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.663075 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.665709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.663784 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.665709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.664078 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.673214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.673079 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.681444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.681337 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.753146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.753116 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tph6p\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.762172 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.762143 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.836805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.836729 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp"] Nov 03 20:34:52.838510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.838477 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:34:52.841806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.841785 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867016 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867062 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fda5636-bc04-47e6-8e16-01372e832616-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867109 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xtzk\" (UniqueName: \"kubernetes.io/projected/5fda5636-bc04-47e6-8e16-01372e832616-kube-api-access-4xtzk\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867159 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fda5636-bc04-47e6-8e16-01372e832616-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867202 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867240 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867266 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867311 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.867422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.867347 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fda5636-bc04-47e6-8e16-01372e832616-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.887708 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.887662 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr"] Nov 03 20:34:52.893867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.892753 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:52.900483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.899820 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp"] Nov 03 20:34:52.927121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.927095 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-tfsgl\"" Nov 03 20:34:52.943163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.941860 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr"] Nov 03 20:34:52.969180 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969040 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xtzk\" (UniqueName: \"kubernetes.io/projected/5fda5636-bc04-47e6-8e16-01372e832616-kube-api-access-4xtzk\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969346 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fda5636-bc04-47e6-8e16-01372e832616-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969369 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:52.969462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969409 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969446 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969559 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969474 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969559 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969519 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969559 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969543 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:52.969748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969614 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqmf\" (UniqueName: \"kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:52.969748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969661 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fda5636-bc04-47e6-8e16-01372e832616-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969708 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.969748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.969737 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fda5636-bc04-47e6-8e16-01372e832616-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.979016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.971533 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fda5636-bc04-47e6-8e16-01372e832616-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.979016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.971841 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.979016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.972137 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.979016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.972749 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:52.979016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.977926 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:53.005651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.996188 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fda5636-bc04-47e6-8e16-01372e832616-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:53.005651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:52.996504 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fda5636-bc04-47e6-8e16-01372e832616-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:53.036252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.033177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xtzk\" (UniqueName: \"kubernetes.io/projected/5fda5636-bc04-47e6-8e16-01372e832616-kube-api-access-4xtzk\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:53.036252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.035459 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fda5636-bc04-47e6-8e16-01372e832616-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp\" (UID: \"5fda5636-bc04-47e6-8e16-01372e832616\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:53.071137 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.071107 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.071425 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.071403 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.071610 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.071592 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqmf\" (UniqueName: \"kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.072422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.072396 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.072838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.072817 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.131230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.131205 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqmf\" (UniqueName: \"kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.162698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.161291 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:34:53.234837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.226181 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:34:53.330751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.330723 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7"] Nov 03 20:34:53.607259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.607161 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp"] Nov 03 20:34:53.654728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:53.654123 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr"] Nov 03 20:34:53.664075 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:34:53.663820 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86145761_e088_48be_89ac_40564a4cd037.slice/crio-cddb908726cc60b56d53131a4206b6c94bffd508b5e87a211eb06b59e005d1fa WatchSource:0}: Error finding container cddb908726cc60b56d53131a4206b6c94bffd508b5e87a211eb06b59e005d1fa: Status 404 returned error can't find the container with id cddb908726cc60b56d53131a4206b6c94bffd508b5e87a211eb06b59e005d1fa Nov 03 20:34:54.315938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:54.315898 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerStarted","Data":"cddb908726cc60b56d53131a4206b6c94bffd508b5e87a211eb06b59e005d1fa"} Nov 03 20:34:54.318348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:54.318314 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" event={"ID":"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017","Type":"ContainerStarted","Data":"ddad01fcc6322ba3bfd855abc1d795e1c1a7a10290d0c95125934a09c94b38d0"} Nov 03 20:34:54.320507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:34:54.320454 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" event={"ID":"5fda5636-bc04-47e6-8e16-01372e832616","Type":"ContainerStarted","Data":"9b735b14e8a678daafbb9a055adb7d32bd412bd3122c71d6caef48dba7f5c530"} Nov 03 20:35:01.417267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:01.417225 5932 generic.go:356] "Generic (PLEG): container finished" podID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerID="217788be51191779c9654923fcc9897cc97e02040941144eaacd01be1196c66a" exitCode=0 Nov 03 20:35:01.417836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:01.417347 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" event={"ID":"b35f0226-f88d-44e2-8ca5-beb89ba661b6","Type":"ContainerDied","Data":"217788be51191779c9654923fcc9897cc97e02040941144eaacd01be1196c66a"} Nov 03 20:35:01.419161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:01.419124 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:35:02.428902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.428863 5932 generic.go:356] "Generic (PLEG): container finished" podID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerID="5456927dc1b8bcb53f7378a1b35f27361c555b363f0d144e9b2158e7acfc13fb" exitCode=0 Nov 03 20:35:02.429391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.428993 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" event={"ID":"92c5a0c0-abe1-470f-85e5-67ec9ad61762","Type":"ContainerDied","Data":"5456927dc1b8bcb53f7378a1b35f27361c555b363f0d144e9b2158e7acfc13fb"} Nov 03 20:35:02.431283 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.431254 5932 generic.go:356] "Generic (PLEG): container finished" podID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerID="cdea7b668128a81999766f7a514236ad7d8dcca2fa78a43f807a0df3e7da5b5a" exitCode=0 Nov 03 20:35:02.431557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.431532 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" event={"ID":"20ef0414-a6cd-4424-8d1b-65fe82502b89","Type":"ContainerDied","Data":"cdea7b668128a81999766f7a514236ad7d8dcca2fa78a43f807a0df3e7da5b5a"} Nov 03 20:35:02.434611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.434568 5932 generic.go:356] "Generic (PLEG): container finished" podID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerID="327341a0649a8c5d17203d2e2572a98ed3526626dd5564a4b33938f618ff1def" exitCode=0 Nov 03 20:35:02.434814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.434764 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" event={"ID":"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467","Type":"ContainerDied","Data":"327341a0649a8c5d17203d2e2572a98ed3526626dd5564a4b33938f618ff1def"} Nov 03 20:35:02.437488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.437464 5932 generic.go:356] "Generic (PLEG): container finished" podID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerID="07ddcca823d4b763345ce3196eadee7478023d00fb5c7bd6ba0c457d3d167822" exitCode=0 Nov 03 20:35:02.437590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.437552 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" event={"ID":"51c552d8-2112-4dd6-ab98-3da570acf7c8","Type":"ContainerDied","Data":"07ddcca823d4b763345ce3196eadee7478023d00fb5c7bd6ba0c457d3d167822"} Nov 03 20:35:02.441252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.441227 5932 generic.go:356] "Generic (PLEG): container finished" podID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerID="25d9cd32af1c7a4bac5ac3aa6b67c2625538735e504cdad1af969ad7f44837e8" exitCode=0 Nov 03 20:35:02.441348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.441286 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" event={"ID":"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2","Type":"ContainerDied","Data":"25d9cd32af1c7a4bac5ac3aa6b67c2625538735e504cdad1af969ad7f44837e8"} Nov 03 20:35:02.781321 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.781283 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:35:02.781432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.781370 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:35:02.781432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.781411 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:35:02.849075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.849030 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:35:02.849189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.849160 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:35:02.849266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:02.849247 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:35:03.448466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.448218 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" event={"ID":"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017","Type":"ContainerStarted","Data":"8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89"} Nov 03 20:35:03.450057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.450019 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" event={"ID":"5fda5636-bc04-47e6-8e16-01372e832616","Type":"ContainerStarted","Data":"a0435c741fe1ca16bb6292e40d09f36761a36640f6c3a43092f5576fac207d65"} Nov 03 20:35:03.451837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.451810 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerStarted","Data":"5a82311f9a635035dbc5426f98bd5ef909ef27f4fb547f72898ac952d9134afc"} Nov 03 20:35:03.562423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.562366 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podStartSLOduration=2.062309502 podStartE2EDuration="11.562346959s" totalImagesPullingTime="9.500037457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:03.560802436 +0000 UTC m=+311.626043910" watchObservedRunningTime="2026-11-03 20:35:03.562346959 +0000 UTC m=+311.627588434" Nov 03 20:35:03.674528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.673478 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" podStartSLOduration=2.518852804 podStartE2EDuration="11.673461747s" totalImagesPullingTime="9.154608943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:03.673081455 +0000 UTC m=+311.738322930" watchObservedRunningTime="2026-11-03 20:35:03.673461747 +0000 UTC m=+311.738703223" Nov 03 20:35:03.838920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.838895 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:35:03.843705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:03.843682 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:35:04.161788 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.161759 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:35:04.168415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.167291 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:35:04.459156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.459089 5932 generic.go:356] "Generic (PLEG): container finished" podID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerID="7c9d9ce5082736eadbf4961265d0d19f0b8ec7431c6e8b3eab0de3649a1eabda" exitCode=0 Nov 03 20:35:04.459615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.459174 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" event={"ID":"b35f0226-f88d-44e2-8ca5-beb89ba661b6","Type":"ContainerDied","Data":"7c9d9ce5082736eadbf4961265d0d19f0b8ec7431c6e8b3eab0de3649a1eabda"} Nov 03 20:35:04.461222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.461192 5932 generic.go:356] "Generic (PLEG): container finished" podID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerID="72c251f96e031c10b412652666b68d0337bc04eae435826a343bdcab75a6b708" exitCode=0 Nov 03 20:35:04.461441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.461418 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" event={"ID":"20ef0414-a6cd-4424-8d1b-65fe82502b89","Type":"ContainerDied","Data":"72c251f96e031c10b412652666b68d0337bc04eae435826a343bdcab75a6b708"} Nov 03 20:35:04.463250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.463139 5932 generic.go:356] "Generic (PLEG): container finished" podID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerID="11fe4324d9dcc26c7672fc7ddc21888252a6d64641304c4e9ce492c4ed4fdf55" exitCode=0 Nov 03 20:35:04.463250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.463195 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" event={"ID":"fa0b12fd-4cdc-421a-8bd5-22228937eab3","Type":"ContainerDied","Data":"11fe4324d9dcc26c7672fc7ddc21888252a6d64641304c4e9ce492c4ed4fdf55"} Nov 03 20:35:04.465370 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.465247 5932 generic.go:356] "Generic (PLEG): container finished" podID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerID="187fc10c1bdda17ccb299fd9103ffe3694e30fa8212bbbefc0662ce7a44b78b3" exitCode=0 Nov 03 20:35:04.465370 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.465322 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" event={"ID":"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467","Type":"ContainerDied","Data":"187fc10c1bdda17ccb299fd9103ffe3694e30fa8212bbbefc0662ce7a44b78b3"} Nov 03 20:35:04.467263 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.467209 5932 generic.go:356] "Generic (PLEG): container finished" podID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerID="81e443087ce3c23ac9072b499b5185ae347d974984f4e80360c1f52315b89028" exitCode=0 Nov 03 20:35:04.467340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.467274 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" event={"ID":"51c552d8-2112-4dd6-ab98-3da570acf7c8","Type":"ContainerDied","Data":"81e443087ce3c23ac9072b499b5185ae347d974984f4e80360c1f52315b89028"} Nov 03 20:35:04.469110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.469092 5932 generic.go:356] "Generic (PLEG): container finished" podID="86145761-e088-48be-89ac-40564a4cd037" containerID="5a82311f9a635035dbc5426f98bd5ef909ef27f4fb547f72898ac952d9134afc" exitCode=0 Nov 03 20:35:04.469183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.469151 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerDied","Data":"5a82311f9a635035dbc5426f98bd5ef909ef27f4fb547f72898ac952d9134afc"} Nov 03 20:35:04.469671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.469654 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:35:04.469923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.469907 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:35:04.470703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.470686 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp" Nov 03 20:35:04.470920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:04.470903 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:35:05.478237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:05.478197 5932 generic.go:356] "Generic (PLEG): container finished" podID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerID="fd1b96409e8a0d87531c9ed646cd9c965faf2b627fa007ac9506078950b00c1b" exitCode=0 Nov 03 20:35:05.478725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:05.478298 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" event={"ID":"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2","Type":"ContainerDied","Data":"fd1b96409e8a0d87531c9ed646cd9c965faf2b627fa007ac9506078950b00c1b"} Nov 03 20:35:05.480671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:05.480650 5932 generic.go:356] "Generic (PLEG): container finished" podID="86145761-e088-48be-89ac-40564a4cd037" containerID="13d4c484952fb115558a0227c338bbeb075269df8a32ce50e75043c41d0df83b" exitCode=0 Nov 03 20:35:05.480783 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:05.480760 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerDied","Data":"13d4c484952fb115558a0227c338bbeb075269df8a32ce50e75043c41d0df83b"} Nov 03 20:35:06.494831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:06.494794 5932 generic.go:356] "Generic (PLEG): container finished" podID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerID="d1feb0ca11af7280c1b117cb34e064b63d3f442567c00d3013e9b25c3114618b" exitCode=0 Nov 03 20:35:06.496027 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:06.495988 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" event={"ID":"92c5a0c0-abe1-470f-85e5-67ec9ad61762","Type":"ContainerDied","Data":"d1feb0ca11af7280c1b117cb34e064b63d3f442567c00d3013e9b25c3114618b"} Nov 03 20:35:12.577725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:12.577684 5932 generic.go:356] "Generic (PLEG): container finished" podID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerID="977f987c92c68c533912c41922d3d61140ba35808e95d6e2bf0dca2999a19294" exitCode=0 Nov 03 20:35:12.578322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:12.577743 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" event={"ID":"fa0b12fd-4cdc-421a-8bd5-22228937eab3","Type":"ContainerDied","Data":"977f987c92c68c533912c41922d3d61140ba35808e95d6e2bf0dca2999a19294"} Nov 03 20:35:14.172096 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:35:14.172050 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:35:21.666725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:21.666665 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" event={"ID":"b35f0226-f88d-44e2-8ca5-beb89ba661b6","Type":"ContainerStarted","Data":"907d29ccb97ec64ea4fe1e127f32834e40d9b5c6e2a9de4c0eedd7e1bee2a170"} Nov 03 20:35:21.669318 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:21.669290 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" event={"ID":"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467","Type":"ContainerStarted","Data":"07c9d1241c41caa975c10e53aa5d10b507eb487d5fe59ba29c6bca357ef45833"} Nov 03 20:35:21.673979 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:21.673950 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" event={"ID":"51c552d8-2112-4dd6-ab98-3da570acf7c8","Type":"ContainerStarted","Data":"23592d0a37625d80126bbcd1964ee3f3cdd8ed14b1e8dfb063f9081eb76068e9"} Nov 03 20:35:21.798733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:21.798607 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" podStartSLOduration=3.183592638 podStartE2EDuration="36.798588864s" totalImagesPullingTime="33.614996226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:21.797875632 +0000 UTC m=+329.863117106" watchObservedRunningTime="2026-11-03 20:35:21.798588864 +0000 UTC m=+329.863830340" Nov 03 20:35:21.908262 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:21.908212 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" podStartSLOduration=3.050558074 podStartE2EDuration="37.908198353s" totalImagesPullingTime="34.857640279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:21.907505241 +0000 UTC m=+329.972746728" watchObservedRunningTime="2026-11-03 20:35:21.908198353 +0000 UTC m=+329.973439826" Nov 03 20:35:22.682158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.682124 5932 generic.go:356] "Generic (PLEG): container finished" podID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerID="907d29ccb97ec64ea4fe1e127f32834e40d9b5c6e2a9de4c0eedd7e1bee2a170" exitCode=0 Nov 03 20:35:22.682535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.682233 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" event={"ID":"b35f0226-f88d-44e2-8ca5-beb89ba661b6","Type":"ContainerDied","Data":"907d29ccb97ec64ea4fe1e127f32834e40d9b5c6e2a9de4c0eedd7e1bee2a170"} Nov 03 20:35:22.687758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.687675 5932 generic.go:356] "Generic (PLEG): container finished" podID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerID="ba9a293c96e1fd36830c3f52b794d49a1db0226fa52b1868e7c75d859a5c8943" exitCode=0 Nov 03 20:35:22.687758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.687707 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" event={"ID":"20ef0414-a6cd-4424-8d1b-65fe82502b89","Type":"ContainerDied","Data":"ba9a293c96e1fd36830c3f52b794d49a1db0226fa52b1868e7c75d859a5c8943"} Nov 03 20:35:22.693036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.690312 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" event={"ID":"fa0b12fd-4cdc-421a-8bd5-22228937eab3","Type":"ContainerStarted","Data":"9dac4c08d657d1c835ebd6bd1f39463fcd7d5bbd1f31966406e8f0444658bb88"} Nov 03 20:35:22.693036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.692893 5932 generic.go:356] "Generic (PLEG): container finished" podID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerID="07c9d1241c41caa975c10e53aa5d10b507eb487d5fe59ba29c6bca357ef45833" exitCode=0 Nov 03 20:35:22.693036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.692949 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" event={"ID":"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467","Type":"ContainerDied","Data":"07c9d1241c41caa975c10e53aa5d10b507eb487d5fe59ba29c6bca357ef45833"} Nov 03 20:35:22.695562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.695532 5932 generic.go:356] "Generic (PLEG): container finished" podID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerID="23592d0a37625d80126bbcd1964ee3f3cdd8ed14b1e8dfb063f9081eb76068e9" exitCode=0 Nov 03 20:35:22.695681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.695651 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" event={"ID":"51c552d8-2112-4dd6-ab98-3da570acf7c8","Type":"ContainerDied","Data":"23592d0a37625d80126bbcd1964ee3f3cdd8ed14b1e8dfb063f9081eb76068e9"} Nov 03 20:35:22.698178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.698155 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerStarted","Data":"e098e529c99e6b4bacccdde729a8e713646ded3b8f187b3b48cbb6d3e5025e82"} Nov 03 20:35:22.913832 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:22.913774 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" podStartSLOduration=3.257702212 podStartE2EDuration="35.913753504s" totalImagesPullingTime="32.656051292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:22.913122398 +0000 UTC m=+330.978363872" watchObservedRunningTime="2026-11-03 20:35:22.913753504 +0000 UTC m=+330.978994979" Nov 03 20:35:23.353956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.353907 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" podStartSLOduration=4.889630534 podStartE2EDuration="31.353892624s" totalImagesPullingTime="26.46426209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:23.352408076 +0000 UTC m=+331.417649549" watchObservedRunningTime="2026-11-03 20:35:23.353892624 +0000 UTC m=+331.419134097" Nov 03 20:35:23.704242 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.704197 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" event={"ID":"92c5a0c0-abe1-470f-85e5-67ec9ad61762","Type":"ContainerStarted","Data":"a14fdc31032e64d43a81c2df685b7e33d808d8e1f5e61a81b3ce9b44c2ac5070"} Nov 03 20:35:23.706232 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.706205 5932 generic.go:356] "Generic (PLEG): container finished" podID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerID="9dac4c08d657d1c835ebd6bd1f39463fcd7d5bbd1f31966406e8f0444658bb88" exitCode=0 Nov 03 20:35:23.706348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.706290 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" event={"ID":"fa0b12fd-4cdc-421a-8bd5-22228937eab3","Type":"ContainerDied","Data":"9dac4c08d657d1c835ebd6bd1f39463fcd7d5bbd1f31966406e8f0444658bb88"} Nov 03 20:35:23.708178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.708159 5932 generic.go:356] "Generic (PLEG): container finished" podID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerID="d725f7a8c4450d22fc5fea55d5bd3b2a7db3f9b69da260a1bdb8a6ff887c9e95" exitCode=0 Nov 03 20:35:23.708282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.708227 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" event={"ID":"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2","Type":"ContainerDied","Data":"d725f7a8c4450d22fc5fea55d5bd3b2a7db3f9b69da260a1bdb8a6ff887c9e95"} Nov 03 20:35:23.710118 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.710100 5932 generic.go:356] "Generic (PLEG): container finished" podID="86145761-e088-48be-89ac-40564a4cd037" containerID="e098e529c99e6b4bacccdde729a8e713646ded3b8f187b3b48cbb6d3e5025e82" exitCode=0 Nov 03 20:35:23.710203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.710185 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerDied","Data":"e098e529c99e6b4bacccdde729a8e713646ded3b8f187b3b48cbb6d3e5025e82"} Nov 03 20:35:23.846120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.846093 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:35:23.927117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.927091 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:35:23.927661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.927602 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" podStartSLOduration=3.106761321 podStartE2EDuration="34.927586811s" totalImagesPullingTime="31.82082549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:23.816112715 +0000 UTC m=+331.881354187" watchObservedRunningTime="2026-11-03 20:35:23.927586811 +0000 UTC m=+331.992828283" Nov 03 20:35:23.930425 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.930406 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:35:23.933694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:23.933679 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:35:24.000812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.000755 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle\") pod \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " Nov 03 20:35:24.000812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.000800 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s\" (UniqueName: \"kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s\") pod \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " Nov 03 20:35:24.000982 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.000860 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util\") pod \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\" (UID: \"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467\") " Nov 03 20:35:24.001280 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.001255 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle" pod "1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" (UID: "1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.002985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.002965 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s" pod "1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" (UID: "1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467"). InnerVolumeSpecName "kube-api-access-fnl7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:24.006304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.006283 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util" pod "1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" (UID: "1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.101292 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101272 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util\") pod \"20ef0414-a6cd-4424-8d1b-65fe82502b89\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " Nov 03 20:35:24.101380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101334 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util\") pod \"51c552d8-2112-4dd6-ab98-3da570acf7c8\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " Nov 03 20:35:24.101380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101365 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util\") pod \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " Nov 03 20:35:24.101460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101395 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle\") pod \"51c552d8-2112-4dd6-ab98-3da570acf7c8\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " Nov 03 20:35:24.101460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101428 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2\" (UniqueName: \"kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2\") pod \"20ef0414-a6cd-4424-8d1b-65fe82502b89\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " Nov 03 20:35:24.101544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101499 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4\" (UniqueName: \"kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4\") pod \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " Nov 03 20:35:24.101603 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101541 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle\") pod \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\" (UID: \"b35f0226-f88d-44e2-8ca5-beb89ba661b6\") " Nov 03 20:35:24.101603 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.101572 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm\" (UniqueName: \"kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm\") pod \"51c552d8-2112-4dd6-ab98-3da570acf7c8\" (UID: \"51c552d8-2112-4dd6-ab98-3da570acf7c8\") " Nov 03 20:35:24.102060 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.102031 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle" pod "b35f0226-f88d-44e2-8ca5-beb89ba661b6" (UID: "b35f0226-f88d-44e2-8ca5-beb89ba661b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.102147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.102118 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle\") pod \"20ef0414-a6cd-4424-8d1b-65fe82502b89\" (UID: \"20ef0414-a6cd-4424-8d1b-65fe82502b89\") " Nov 03 20:35:24.102232 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.102205 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle" pod "51c552d8-2112-4dd6-ab98-3da570acf7c8" (UID: "51c552d8-2112-4dd6-ab98-3da570acf7c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.102801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.102769 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle" pod "20ef0414-a6cd-4424-8d1b-65fe82502b89" (UID: "20ef0414-a6cd-4424-8d1b-65fe82502b89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.103103 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.103088 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.103211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.103192 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.103211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.103213 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.103387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.103228 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.103387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.103243 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnl7s\" (UniqueName: \"kubernetes.io/projected/1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467-kube-api-access-fnl7s\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.103387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.103255 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.104171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.104141 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4" pod "b35f0226-f88d-44e2-8ca5-beb89ba661b6" (UID: "b35f0226-f88d-44e2-8ca5-beb89ba661b6"). InnerVolumeSpecName "kube-api-access-khhm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:24.104259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.104150 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2" pod "20ef0414-a6cd-4424-8d1b-65fe82502b89" (UID: "20ef0414-a6cd-4424-8d1b-65fe82502b89"). InnerVolumeSpecName "kube-api-access-v27q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:24.104259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.104195 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm" pod "51c552d8-2112-4dd6-ab98-3da570acf7c8" (UID: "51c552d8-2112-4dd6-ab98-3da570acf7c8"). InnerVolumeSpecName "kube-api-access-cmhbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:24.109232 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.109213 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util" pod "51c552d8-2112-4dd6-ab98-3da570acf7c8" (UID: "51c552d8-2112-4dd6-ab98-3da570acf7c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.109388 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.109364 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util" pod "20ef0414-a6cd-4424-8d1b-65fe82502b89" (UID: "20ef0414-a6cd-4424-8d1b-65fe82502b89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.109654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.109635 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util" pod "b35f0226-f88d-44e2-8ca5-beb89ba661b6" (UID: "b35f0226-f88d-44e2-8ca5-beb89ba661b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.204435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.204408 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51c552d8-2112-4dd6-ab98-3da570acf7c8-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.204435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.204430 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b35f0226-f88d-44e2-8ca5-beb89ba661b6-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.204587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.204440 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v27q2\" (UniqueName: \"kubernetes.io/projected/20ef0414-a6cd-4424-8d1b-65fe82502b89-kube-api-access-v27q2\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.204587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.204456 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhm4\" (UniqueName: \"kubernetes.io/projected/b35f0226-f88d-44e2-8ca5-beb89ba661b6-kube-api-access-khhm4\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.204587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.204465 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmhbm\" (UniqueName: \"kubernetes.io/projected/51c552d8-2112-4dd6-ab98-3da570acf7c8-kube-api-access-cmhbm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.204587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.204474 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/20ef0414-a6cd-4424-8d1b-65fe82502b89-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:24.716220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.716181 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" event={"ID":"b35f0226-f88d-44e2-8ca5-beb89ba661b6","Type":"ContainerDied","Data":"859077a428b745d583c68445e2d59d49347ccd56a4f0d918001e777ad1bbee5d"} Nov 03 20:35:24.716682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.716236 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="859077a428b745d583c68445e2d59d49347ccd56a4f0d918001e777ad1bbee5d" Nov 03 20:35:24.716682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.716191 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6wdzf5" Nov 03 20:35:24.718273 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.718249 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" Nov 03 20:35:24.718392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.718249 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nrpcx" event={"ID":"20ef0414-a6cd-4424-8d1b-65fe82502b89","Type":"ContainerDied","Data":"f9b9c4effbae6058f510ad9216148df287987a09a91a0a9c21698315f1290e3d"} Nov 03 20:35:24.718392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.718383 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b9c4effbae6058f510ad9216148df287987a09a91a0a9c21698315f1290e3d" Nov 03 20:35:24.720192 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.720165 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" event={"ID":"1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467","Type":"ContainerDied","Data":"a88502dd3be9cbfc7d6109adbce0832caea6dfce0428d1d555ccdee3f1694311"} Nov 03 20:35:24.720293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.720205 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a88502dd3be9cbfc7d6109adbce0832caea6dfce0428d1d555ccdee3f1694311" Nov 03 20:35:24.720293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.720171 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mgtmt" Nov 03 20:35:24.722855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.722581 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" Nov 03 20:35:24.722855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.722593 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a7l7fr" event={"ID":"51c552d8-2112-4dd6-ab98-3da570acf7c8","Type":"ContainerDied","Data":"ae931cf80f8860f0cbd9d53d67456f3ab19e85f0abeedbbeaabb9cb4a0e9be5b"} Nov 03 20:35:24.722855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.722658 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae931cf80f8860f0cbd9d53d67456f3ab19e85f0abeedbbeaabb9cb4a0e9be5b" Nov 03 20:35:24.724925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.724902 5932 generic.go:356] "Generic (PLEG): container finished" podID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerID="a14fdc31032e64d43a81c2df685b7e33d808d8e1f5e61a81b3ce9b44c2ac5070" exitCode=0 Nov 03 20:35:24.725012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.724935 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" event={"ID":"92c5a0c0-abe1-470f-85e5-67ec9ad61762","Type":"ContainerDied","Data":"a14fdc31032e64d43a81c2df685b7e33d808d8e1f5e61a81b3ce9b44c2ac5070"} Nov 03 20:35:24.887645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.887596 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:35:24.904286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.904264 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:35:24.909461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.909438 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb\" (UniqueName: \"kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb\") pod \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " Nov 03 20:35:24.909557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.909495 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf\" (UniqueName: \"kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf\") pod \"86145761-e088-48be-89ac-40564a4cd037\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " Nov 03 20:35:24.909603 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.909558 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle\") pod \"86145761-e088-48be-89ac-40564a4cd037\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " Nov 03 20:35:24.909721 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.909703 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util\") pod \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " Nov 03 20:35:24.909785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.909752 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util\") pod \"86145761-e088-48be-89ac-40564a4cd037\" (UID: \"86145761-e088-48be-89ac-40564a4cd037\") " Nov 03 20:35:24.909839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.909794 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle\") pod \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\" (UID: \"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2\") " Nov 03 20:35:24.910308 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.910281 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle" pod "86145761-e088-48be-89ac-40564a4cd037" (UID: "86145761-e088-48be-89ac-40564a4cd037"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.910484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.910461 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle" pod "c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" (UID: "c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.911972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.911945 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf" pod "86145761-e088-48be-89ac-40564a4cd037" (UID: "86145761-e088-48be-89ac-40564a4cd037"). InnerVolumeSpecName "kube-api-access-wkqmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:24.912157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.912141 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb" pod "c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" (UID: "c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2"). InnerVolumeSpecName "kube-api-access-8ldgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:24.917672 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.917645 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util" pod "c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" (UID: "c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.917970 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.917953 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util" pod "86145761-e088-48be-89ac-40564a4cd037" (UID: "86145761-e088-48be-89ac-40564a4cd037"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:24.940401 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:24.940384 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:35:25.010368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010303 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util\") pod \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " Nov 03 20:35:25.010368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010337 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle\") pod \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " Nov 03 20:35:25.010571 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010374 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p\" (UniqueName: \"kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p\") pod \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\" (UID: \"fa0b12fd-4cdc-421a-8bd5-22228937eab3\") " Nov 03 20:35:25.010667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010651 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ldgb\" (UniqueName: \"kubernetes.io/projected/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-kube-api-access-8ldgb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.010734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010675 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqmf\" (UniqueName: \"kubernetes.io/projected/86145761-e088-48be-89ac-40564a4cd037-kube-api-access-wkqmf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.010734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010691 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.010734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010705 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.010734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010718 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86145761-e088-48be-89ac-40564a4cd037-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.010734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010732 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.010914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.010755 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle" pod "fa0b12fd-4cdc-421a-8bd5-22228937eab3" (UID: "fa0b12fd-4cdc-421a-8bd5-22228937eab3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.012415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.012394 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p" pod "fa0b12fd-4cdc-421a-8bd5-22228937eab3" (UID: "fa0b12fd-4cdc-421a-8bd5-22228937eab3"). InnerVolumeSpecName "kube-api-access-rhd9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:25.014904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.014882 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util" pod "fa0b12fd-4cdc-421a-8bd5-22228937eab3" (UID: "fa0b12fd-4cdc-421a-8bd5-22228937eab3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.116191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.111437 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.116191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.111479 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa0b12fd-4cdc-421a-8bd5-22228937eab3-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.116191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.111496 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhd9p\" (UniqueName: \"kubernetes.io/projected/fa0b12fd-4cdc-421a-8bd5-22228937eab3-kube-api-access-rhd9p\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:25.731591 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.731554 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" event={"ID":"fa0b12fd-4cdc-421a-8bd5-22228937eab3","Type":"ContainerDied","Data":"4c744f4a8e4e7a7cdf9104f1f8bbfe957d6aae2ed27ed33dd21b37e4935027b6"} Nov 03 20:35:25.732017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.731571 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644f4qcj" Nov 03 20:35:25.732017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.731668 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c744f4a8e4e7a7cdf9104f1f8bbfe957d6aae2ed27ed33dd21b37e4935027b6" Nov 03 20:35:25.733543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.733521 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" Nov 03 20:35:25.733661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.733515 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdpnxv" event={"ID":"c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2","Type":"ContainerDied","Data":"f50ab674dac85a3cddaaea51285ceef22998bf7fcbaffb3e25732ce3fb188597"} Nov 03 20:35:25.733661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.733650 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50ab674dac85a3cddaaea51285ceef22998bf7fcbaffb3e25732ce3fb188597" Nov 03 20:35:25.735611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.735583 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" event={"ID":"86145761-e088-48be-89ac-40564a4cd037","Type":"ContainerDied","Data":"cddb908726cc60b56d53131a4206b6c94bffd508b5e87a211eb06b59e005d1fa"} Nov 03 20:35:25.735718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.735645 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7m9hzr" Nov 03 20:35:25.735718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.735644 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddb908726cc60b56d53131a4206b6c94bffd508b5e87a211eb06b59e005d1fa" Nov 03 20:35:25.866464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.866440 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:35:25.916661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.916614 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w\" (UniqueName: \"kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w\") pod \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " Nov 03 20:35:25.916821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.916705 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle\") pod \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " Nov 03 20:35:25.916821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.916743 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util\") pod \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\" (UID: \"92c5a0c0-abe1-470f-85e5-67ec9ad61762\") " Nov 03 20:35:25.917367 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.917336 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle" pod "92c5a0c0-abe1-470f-85e5-67ec9ad61762" (UID: "92c5a0c0-abe1-470f-85e5-67ec9ad61762"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:25.918860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.918830 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w" pod "92c5a0c0-abe1-470f-85e5-67ec9ad61762" (UID: "92c5a0c0-abe1-470f-85e5-67ec9ad61762"). InnerVolumeSpecName "kube-api-access-92q4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:25.921183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:25.921164 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util" pod "92c5a0c0-abe1-470f-85e5-67ec9ad61762" (UID: "92c5a0c0-abe1-470f-85e5-67ec9ad61762"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:26.017917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:26.017844 5932 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-util\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:26.017917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:26.017871 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92q4w\" (UniqueName: \"kubernetes.io/projected/92c5a0c0-abe1-470f-85e5-67ec9ad61762-kube-api-access-92q4w\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:26.017917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:26.017882 5932 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/92c5a0c0-abe1-470f-85e5-67ec9ad61762-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:26.742464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:26.742426 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" event={"ID":"92c5a0c0-abe1-470f-85e5-67ec9ad61762","Type":"ContainerDied","Data":"fb4b8582ccef73b4fb2914732d527d7ebe009a333e0dd1081db3c1214be80479"} Nov 03 20:35:26.742464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:26.742464 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c75c9s" Nov 03 20:35:26.742464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:26.742474 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4b8582ccef73b4fb2914732d527d7ebe009a333e0dd1081db3c1214be80479" Nov 03 20:35:30.948295 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948258 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6"] Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948767 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="util" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948786 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="util" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948802 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="pull" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948810 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="pull" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948823 5932 cpu_manager.go:414] "containerMap: removing container" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="util" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948830 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="util" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948841 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="pull" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948847 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="pull" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948856 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="pull" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948863 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="pull" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948875 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="util" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948881 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="util" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948891 5932 cpu_manager.go:414] "containerMap: removing container" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="extract" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948897 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="extract" Nov 03 20:35:30.948907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948910 5932 cpu_manager.go:414] "containerMap: removing container" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948917 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948932 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948938 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948947 5932 cpu_manager.go:414] "containerMap: removing container" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948955 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948970 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948976 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948986 5932 cpu_manager.go:414] "containerMap: removing container" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.948992 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949003 5932 cpu_manager.go:414] "containerMap: removing container" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949010 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949019 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949026 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949037 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949044 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949055 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949062 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949072 5932 cpu_manager.go:414] "containerMap: removing container" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949079 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949088 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949095 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949105 5932 cpu_manager.go:414] "containerMap: removing container" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949112 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949129 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949135 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949146 5932 cpu_manager.go:414] "containerMap: removing container" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949153 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="util" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949165 5932 cpu_manager.go:414] "containerMap: removing container" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949172 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949180 5932 cpu_manager.go:414] "containerMap: removing container" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949187 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949198 5932 cpu_manager.go:414] "containerMap: removing container" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949206 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="pull" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949290 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c552d8-2112-4dd6-ab98-3da570acf7c8" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949303 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b35f0226-f88d-44e2-8ca5-beb89ba661b6" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949313 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e21d21-8bc2-4e0f-a658-9fc0d5be86e2" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949321 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="92c5a0c0-abe1-470f-85e5-67ec9ad61762" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949331 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ef0414-a6cd-4424-8d1b-65fe82502b89" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949339 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f773a6e-5bab-4b86-9eb5-8e8d0d9ed467" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949347 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0b12fd-4cdc-421a-8bd5-22228937eab3" containerName="extract" Nov 03 20:35:30.949652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.949357 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="86145761-e088-48be-89ac-40564a4cd037" containerName="extract" Nov 03 20:35:30.953677 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:30.953657 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:31.008454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.008427 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:35:31.008596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.008441 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:35:31.008596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.008505 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 03 20:35:31.008596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.008580 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zphns\"" Nov 03 20:35:31.010193 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.010175 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6"] Nov 03 20:35:31.052774 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.052742 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpmr\" (UniqueName: \"kubernetes.io/projected/141d9b7c-0036-47c1-8299-b0f83f14026e-kube-api-access-xzpmr\") pod \"dns-operator-controller-manager-8557cbdb84-j4td6\" (UID: \"141d9b7c-0036-47c1-8299-b0f83f14026e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:31.153566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.153541 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpmr\" (UniqueName: \"kubernetes.io/projected/141d9b7c-0036-47c1-8299-b0f83f14026e-kube-api-access-xzpmr\") pod \"dns-operator-controller-manager-8557cbdb84-j4td6\" (UID: \"141d9b7c-0036-47c1-8299-b0f83f14026e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:31.210939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.210884 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpmr\" (UniqueName: \"kubernetes.io/projected/141d9b7c-0036-47c1-8299-b0f83f14026e-kube-api-access-xzpmr\") pod \"dns-operator-controller-manager-8557cbdb84-j4td6\" (UID: \"141d9b7c-0036-47c1-8299-b0f83f14026e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:31.266317 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.266288 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:31.636828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.636799 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6"] Nov 03 20:35:31.643404 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:35:31.643374 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141d9b7c_0036_47c1_8299_b0f83f14026e.slice/crio-f1ae683079e9e446731512df4173e4627e502154d03324d0dbdca87ae7537fdc WatchSource:0}: Error finding container f1ae683079e9e446731512df4173e4627e502154d03324d0dbdca87ae7537fdc: Status 404 returned error can't find the container with id f1ae683079e9e446731512df4173e4627e502154d03324d0dbdca87ae7537fdc Nov 03 20:35:31.766095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:31.766066 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" event={"ID":"141d9b7c-0036-47c1-8299-b0f83f14026e","Type":"ContainerStarted","Data":"f1ae683079e9e446731512df4173e4627e502154d03324d0dbdca87ae7537fdc"} Nov 03 20:35:34.435814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.435756 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97"] Nov 03 20:35:34.445947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.445920 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:34.478831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.478799 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wglj\" (UniqueName: \"kubernetes.io/projected/a4eb75db-6377-44f0-8c02-9ed350b94153-kube-api-access-6wglj\") pod \"limitador-operator-controller-manager-5778bb4449-7vr97\" (UID: \"a4eb75db-6377-44f0-8c02-9ed350b94153\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:34.498652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.498448 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97"] Nov 03 20:35:34.544239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.544199 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-9dbqp\"" Nov 03 20:35:34.579197 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.579160 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wglj\" (UniqueName: \"kubernetes.io/projected/a4eb75db-6377-44f0-8c02-9ed350b94153-kube-api-access-6wglj\") pod \"limitador-operator-controller-manager-5778bb4449-7vr97\" (UID: \"a4eb75db-6377-44f0-8c02-9ed350b94153\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:34.636504 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.636477 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wglj\" (UniqueName: \"kubernetes.io/projected/a4eb75db-6377-44f0-8c02-9ed350b94153-kube-api-access-6wglj\") pod \"limitador-operator-controller-manager-5778bb4449-7vr97\" (UID: \"a4eb75db-6377-44f0-8c02-9ed350b94153\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:34.761024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:34.760947 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:35.140473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.140439 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97"] Nov 03 20:35:35.152161 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:35:35.152128 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4eb75db_6377_44f0_8c02_9ed350b94153.slice/crio-d09d4df5beb3db6b6b007c2180182e7e86bd10b44affe73b1ad0ea5ad3cf1c47 WatchSource:0}: Error finding container d09d4df5beb3db6b6b007c2180182e7e86bd10b44affe73b1ad0ea5ad3cf1c47: Status 404 returned error can't find the container with id d09d4df5beb3db6b6b007c2180182e7e86bd10b44affe73b1ad0ea5ad3cf1c47 Nov 03 20:35:35.666286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.666249 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn"] Nov 03 20:35:35.669875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.669859 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.686940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.686919 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed36f878-c834-4e61-9eb1-b97d1de2955c-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.687052 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.686955 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed36f878-c834-4e61-9eb1-b97d1de2955c-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.687052 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.686979 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.687137 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.687080 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.687185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.687149 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtg6\" (UniqueName: \"kubernetes.io/projected/ed36f878-c834-4e61-9eb1-b97d1de2955c-kube-api-access-vgtg6\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.687185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.687173 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.719219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.719196 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:35:35.719360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.719333 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:35:35.719468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.719376 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:35:35.719468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.719440 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:35:35.719715 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.719692 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:35:35.719795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.719768 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-fplxc\"" Nov 03 20:35:35.787495 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787473 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed36f878-c834-4e61-9eb1-b97d1de2955c-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.787654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787514 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed36f878-c834-4e61-9eb1-b97d1de2955c-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.787654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787542 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.787654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787559 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.787654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787599 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtg6\" (UniqueName: \"kubernetes.io/projected/ed36f878-c834-4e61-9eb1-b97d1de2955c-kube-api-access-vgtg6\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.787654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787634 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.787959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.787938 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed36f878-c834-4e61-9eb1-b97d1de2955c-manifests\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.788164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.788141 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" event={"ID":"a4eb75db-6377-44f0-8c02-9ed350b94153","Type":"ContainerStarted","Data":"d09d4df5beb3db6b6b007c2180182e7e86bd10b44affe73b1ad0ea5ad3cf1c47"} Nov 03 20:35:35.790417 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.790393 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" event={"ID":"141d9b7c-0036-47c1-8299-b0f83f14026e","Type":"ContainerStarted","Data":"d8141c23399ec28d841ae03bd612458635029ab900ea0d7104bda84b165502de"} Nov 03 20:35:35.790523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.790426 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.790656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.790612 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:35.790732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.790666 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.790919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.790899 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed36f878-c834-4e61-9eb1-b97d1de2955c-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.791066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.791049 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed36f878-c834-4e61-9eb1-b97d1de2955c-tmp\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.860118 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.860087 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtg6\" (UniqueName: \"kubernetes.io/projected/ed36f878-c834-4e61-9eb1-b97d1de2955c-kube-api-access-vgtg6\") pod \"opendatahub-operator-controller-manager-6f448d84c4-622cn\" (UID: \"ed36f878-c834-4e61-9eb1-b97d1de2955c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:35.924138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.924064 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn"] Nov 03 20:35:35.924318 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.924269 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" podStartSLOduration=2.331102782 podStartE2EDuration="5.924254003s" totalImagesPullingTime="3.593151221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:35.923894049 +0000 UTC m=+343.989135523" watchObservedRunningTime="2026-11-03 20:35:35.924254003 +0000 UTC m=+343.989495478" Nov 03 20:35:35.985310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:35.985283 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:36.525315 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:36.525284 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn"] Nov 03 20:35:36.531569 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:35:36.531543 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded36f878_c834_4e61_9eb1_b97d1de2955c.slice/crio-018b6cc9a6571e829a014b32863bb1fba7036e7ead450b43722fbee8c8795583 WatchSource:0}: Error finding container 018b6cc9a6571e829a014b32863bb1fba7036e7ead450b43722fbee8c8795583: Status 404 returned error can't find the container with id 018b6cc9a6571e829a014b32863bb1fba7036e7ead450b43722fbee8c8795583 Nov 03 20:35:36.797202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:36.797114 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" event={"ID":"ed36f878-c834-4e61-9eb1-b97d1de2955c","Type":"ContainerStarted","Data":"018b6cc9a6571e829a014b32863bb1fba7036e7ead450b43722fbee8c8795583"} Nov 03 20:35:38.810334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:38.810296 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" event={"ID":"a4eb75db-6377-44f0-8c02-9ed350b94153","Type":"ContainerStarted","Data":"3be09d893bd435af9b120056777123752ec0f33a2b2dbd53340d0db54d18f054"} Nov 03 20:35:38.810810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:38.810452 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:38.920197 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:38.920135 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" podStartSLOduration=2.026296243 podStartE2EDuration="4.920116394s" totalImagesPullingTime="2.893820151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:38.919220141 +0000 UTC m=+346.984461615" watchObservedRunningTime="2026-11-03 20:35:38.920116394 +0000 UTC m=+346.985357867" Nov 03 20:35:40.158126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.158099 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m"] Nov 03 20:35:40.163399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.163382 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:40.217467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.217445 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m"] Nov 03 20:35:40.220732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.220715 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6ktjs\"" Nov 03 20:35:40.226988 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.226964 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blzg4\" (UniqueName: \"kubernetes.io/projected/b3bde11f-a135-4286-9b8e-c9c4c1bf3863-kube-api-access-blzg4\") pod \"authorino-operator-7b7bb4bc77-zdz7m\" (UID: \"b3bde11f-a135-4286-9b8e-c9c4c1bf3863\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:40.327434 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.327413 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blzg4\" (UniqueName: \"kubernetes.io/projected/b3bde11f-a135-4286-9b8e-c9c4c1bf3863-kube-api-access-blzg4\") pod \"authorino-operator-7b7bb4bc77-zdz7m\" (UID: \"b3bde11f-a135-4286-9b8e-c9c4c1bf3863\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:40.391773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.391748 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blzg4\" (UniqueName: \"kubernetes.io/projected/b3bde11f-a135-4286-9b8e-c9c4c1bf3863-kube-api-access-blzg4\") pod \"authorino-operator-7b7bb4bc77-zdz7m\" (UID: \"b3bde11f-a135-4286-9b8e-c9c4c1bf3863\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:40.478278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.478254 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:40.822213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.822130 5932 generic.go:356] "Generic (PLEG): container finished" podID="ed36f878-c834-4e61-9eb1-b97d1de2955c" containerID="1bc58f39bc58a79dbd15023663f047513a12138db3772fe375af6aadd3189726" exitCode=0 Nov 03 20:35:40.822362 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.822211 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" event={"ID":"ed36f878-c834-4e61-9eb1-b97d1de2955c","Type":"ContainerDied","Data":"1bc58f39bc58a79dbd15023663f047513a12138db3772fe375af6aadd3189726"} Nov 03 20:35:40.933793 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:40.933758 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m"] Nov 03 20:35:40.937674 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:35:40.937646 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bde11f_a135_4286_9b8e_c9c4c1bf3863.slice/crio-25968dac3cde1ff630794b690b2de6e2d6010f2e89bb648059a43274e55e5082 WatchSource:0}: Error finding container 25968dac3cde1ff630794b690b2de6e2d6010f2e89bb648059a43274e55e5082: Status 404 returned error can't find the container with id 25968dac3cde1ff630794b690b2de6e2d6010f2e89bb648059a43274e55e5082 Nov 03 20:35:41.828566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:41.828528 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" event={"ID":"b3bde11f-a135-4286-9b8e-c9c4c1bf3863","Type":"ContainerStarted","Data":"25968dac3cde1ff630794b690b2de6e2d6010f2e89bb648059a43274e55e5082"} Nov 03 20:35:41.830487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:41.830462 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" event={"ID":"ed36f878-c834-4e61-9eb1-b97d1de2955c","Type":"ContainerStarted","Data":"4715fff5e60bccf8c27cd68befbd235f131a76342e090da37f9c125e7c29cad8"} Nov 03 20:35:41.830666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:41.830648 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:42.057673 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:42.057569 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" podStartSLOduration=2.141294733 podStartE2EDuration="7.057556867s" totalImagesPullingTime="4.461929185s" totalInitContainerRuntime="454.332949ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:42.01143737 +0000 UTC m=+350.076678843" watchObservedRunningTime="2026-11-03 20:35:42.057556867 +0000 UTC m=+350.122798339" Nov 03 20:35:44.148265 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:35:44.148231 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:35:44.852562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:44.852527 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" event={"ID":"b3bde11f-a135-4286-9b8e-c9c4c1bf3863","Type":"ContainerStarted","Data":"d23bed0517197636c70d461d9b5a1ab1eb11eb9fdb20e7ce373c0b02d021e2f8"} Nov 03 20:35:44.852759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:44.852589 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:44.964012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:44.963963 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" podStartSLOduration=2.067322135 podStartE2EDuration="4.963948917s" totalImagesPullingTime="2.896626782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:44.962252752 +0000 UTC m=+353.027494224" watchObservedRunningTime="2026-11-03 20:35:44.963948917 +0000 UTC m=+353.029190386" Nov 03 20:35:46.800065 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:46.800032 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-j4td6" Nov 03 20:35:49.817890 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:49.817864 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-7vr97" Nov 03 20:35:51.996753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:51.996720 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg"] Nov 03 20:35:52.000486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.000468 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.023480 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.023458 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b82871c-415e-4192-bdc0-2f8a77511042-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg\" (UID: \"6b82871c-415e-4192-bdc0-2f8a77511042\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.023597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.023530 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9j2\" (UniqueName: \"kubernetes.io/projected/6b82871c-415e-4192-bdc0-2f8a77511042-kube-api-access-lm9j2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg\" (UID: \"6b82871c-415e-4192-bdc0-2f8a77511042\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.099931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.099904 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:35:52.100086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.099937 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:35:52.100086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.100025 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5m59r\"" Nov 03 20:35:52.101459 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.101440 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg"] Nov 03 20:35:52.124363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.124336 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b82871c-415e-4192-bdc0-2f8a77511042-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg\" (UID: \"6b82871c-415e-4192-bdc0-2f8a77511042\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.124474 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.124407 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9j2\" (UniqueName: \"kubernetes.io/projected/6b82871c-415e-4192-bdc0-2f8a77511042-kube-api-access-lm9j2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg\" (UID: \"6b82871c-415e-4192-bdc0-2f8a77511042\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.124751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.124735 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6b82871c-415e-4192-bdc0-2f8a77511042-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg\" (UID: \"6b82871c-415e-4192-bdc0-2f8a77511042\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.181353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.181327 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9j2\" (UniqueName: \"kubernetes.io/projected/6b82871c-415e-4192-bdc0-2f8a77511042-kube-api-access-lm9j2\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg\" (UID: \"6b82871c-415e-4192-bdc0-2f8a77511042\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.311014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.310957 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" Nov 03 20:35:52.781148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.781110 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg"] Nov 03 20:35:52.787873 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:35:52.787843 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b82871c_415e_4192_bdc0_2f8a77511042.slice/crio-e446f12a80808d80110914db3048181b6c5a88e7db4ccf8f43fe48aa63f52833 WatchSource:0}: Error finding container e446f12a80808d80110914db3048181b6c5a88e7db4ccf8f43fe48aa63f52833: Status 404 returned error can't find the container with id e446f12a80808d80110914db3048181b6c5a88e7db4ccf8f43fe48aa63f52833 Nov 03 20:35:52.843340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.843318 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f448d84c4-622cn" Nov 03 20:35:52.893256 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:52.893221 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" event={"ID":"6b82871c-415e-4192-bdc0-2f8a77511042","Type":"ContainerStarted","Data":"e446f12a80808d80110914db3048181b6c5a88e7db4ccf8f43fe48aa63f52833"} Nov 03 20:35:55.860073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:55.859997 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zdz7m" Nov 03 20:35:55.909984 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:55.909948 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" event={"ID":"6b82871c-415e-4192-bdc0-2f8a77511042","Type":"ContainerStarted","Data":"7208d8e749a3ac95e4f09d198fc7c5e68713f9f3ad403410fe59ddc399f42363"} Nov 03 20:35:56.107018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:35:56.106961 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h9mqg" podStartSLOduration=2.321522345 podStartE2EDuration="5.106948036s" totalImagesPullingTime="2.785425691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:35:56.106242942 +0000 UTC m=+364.171484415" watchObservedRunningTime="2026-11-03 20:35:56.106948036 +0000 UTC m=+364.172189508" Nov 03 20:36:14.149454 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:36:14.149368 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:36:17.660583 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.660550 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg"] Nov 03 20:36:17.672567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.672545 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.723138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.723114 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg"] Nov 03 20:36:17.746009 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.745979 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746023 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746054 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746232 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746178 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746266 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746318 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746357 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746392 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk285\" (UniqueName: \"kubernetes.io/projected/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-kube-api-access-xk285\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.746457 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.746430 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.767537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.767513 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-d5n9g\"" Nov 03 20:36:17.847566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.847537 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.847566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.847573 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.847823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.847593 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk285\" (UniqueName: \"kubernetes.io/projected/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-kube-api-access-xk285\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.847823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.847669 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.847976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.847798 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.847995 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848024 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848151 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848063 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848151 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848140 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848160 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848242 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848397 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848429 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.848543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.848493 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.850482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.850457 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.850780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.850761 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.904791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.904768 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.904893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.904839 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk285\" (UniqueName: \"kubernetes.io/projected/f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e-kube-api-access-xk285\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg\" (UID: \"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:17.985400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:17.985330 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:18.407636 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:18.407590 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg"] Nov 03 20:36:18.414334 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:36:18.414310 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13d4fa7_4c1e_42fe_9ad5_d6c82f84065e.slice/crio-cdf104ba9c9f94a2281a2f89eb3c92253c7d4cf7d11fe339ebad94ed9302745f WatchSource:0}: Error finding container cdf104ba9c9f94a2281a2f89eb3c92253c7d4cf7d11fe339ebad94ed9302745f: Status 404 returned error can't find the container with id cdf104ba9c9f94a2281a2f89eb3c92253c7d4cf7d11fe339ebad94ed9302745f Nov 03 20:36:18.416967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:18.416933 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:36:18.417062 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:18.417012 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:36:18.417120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:18.417062 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:36:19.026662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.026615 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" event={"ID":"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e","Type":"ContainerStarted","Data":"f329304e0ba12b7a96bf1bcb14545067f982e99a3a9b829aeede2886d2fbda27"} Nov 03 20:36:19.027044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.026712 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" event={"ID":"f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e","Type":"ContainerStarted","Data":"cdf104ba9c9f94a2281a2f89eb3c92253c7d4cf7d11fe339ebad94ed9302745f"} Nov 03 20:36:19.186550 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.186482 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" podStartSLOduration=2.186469777 podStartE2EDuration="2.186469777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:19.184915247 +0000 UTC m=+387.250156722" watchObservedRunningTime="2026-11-03 20:36:19.186469777 +0000 UTC m=+387.251711255" Nov 03 20:36:19.526386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.526360 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:19.530660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.530643 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.581644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.581594 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tfjxg\"" Nov 03 20:36:19.581785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.581763 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:36:19.583397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.583366 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:19.665034 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.665003 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qd9\" (UniqueName: \"kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9\") pod \"limitador-limitador-74dfcfb894-vcwsd\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.665279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.665053 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file\") pod \"limitador-limitador-74dfcfb894-vcwsd\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.764102 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.764058 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:19.766403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.766377 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file\") pod \"limitador-limitador-74dfcfb894-vcwsd\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.766520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.766492 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qd9\" (UniqueName: \"kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9\") pod \"limitador-limitador-74dfcfb894-vcwsd\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.767101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.767083 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file\") pod \"limitador-limitador-74dfcfb894-vcwsd\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.826850 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.826789 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qd9\" (UniqueName: \"kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9\") pod \"limitador-limitador-74dfcfb894-vcwsd\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.843743 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.843719 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:19.986159 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.986130 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:19.990833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:19.990811 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:20.032265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.032239 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:20.033121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.033104 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg" Nov 03 20:36:20.221385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.221360 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:20.227513 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:36:20.227488 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec367157_0297_4848_8328_d86c512654f5.slice/crio-e6377042a52c8d5e803204742ce6c63fd58fc017b855d76ed7945f6a3e8cd370 WatchSource:0}: Error finding container e6377042a52c8d5e803204742ce6c63fd58fc017b855d76ed7945f6a3e8cd370: Status 404 returned error can't find the container with id e6377042a52c8d5e803204742ce6c63fd58fc017b855d76ed7945f6a3e8cd370 Nov 03 20:36:20.894114 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.894078 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4jh69"] Nov 03 20:36:20.898737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.898718 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:20.950014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.949984 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4jh69"] Nov 03 20:36:20.965581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:20.965549 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4v8fx\"" Nov 03 20:36:21.038137 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:21.038103 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" event={"ID":"ec367157-0297-4848-8328-d86c512654f5","Type":"ContainerStarted","Data":"e6377042a52c8d5e803204742ce6c63fd58fc017b855d76ed7945f6a3e8cd370"} Nov 03 20:36:21.078091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:21.078065 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9kt\" (UniqueName: \"kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt\") pod \"authorino-57f7f8659f-4jh69\" (UID: \"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59\") " pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:21.179241 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:21.179148 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9kt\" (UniqueName: \"kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt\") pod \"authorino-57f7f8659f-4jh69\" (UID: \"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59\") " pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:21.237016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:21.236991 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9kt\" (UniqueName: \"kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt\") pod \"authorino-57f7f8659f-4jh69\" (UID: \"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59\") " pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:21.521804 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:21.521733 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:21.886334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:21.886304 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4jh69"] Nov 03 20:36:21.892829 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:36:21.892800 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82942f9f_1d21_4d3f_aaf5_b5c1d93d2c59.slice/crio-d8bfe0cd1b698fc632bacad86ed3247c9eec2c9595e4d2fe205150e0c208ba47 WatchSource:0}: Error finding container d8bfe0cd1b698fc632bacad86ed3247c9eec2c9595e4d2fe205150e0c208ba47: Status 404 returned error can't find the container with id d8bfe0cd1b698fc632bacad86ed3247c9eec2c9595e4d2fe205150e0c208ba47 Nov 03 20:36:22.050411 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.050380 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4jh69" event={"ID":"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59","Type":"ContainerStarted","Data":"d8bfe0cd1b698fc632bacad86ed3247c9eec2c9595e4d2fe205150e0c208ba47"} Nov 03 20:36:22.698709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.698675 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-86z67"] Nov 03 20:36:22.702267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.702245 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:36:22.763549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.763520 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-86z67"] Nov 03 20:36:22.791088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.791063 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7xh\" (UniqueName: \"kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh\") pod \"authorino-56c6c9ddd5-86z67\" (UID: \"e4f3dcbf-d30a-4151-9721-78d03fe90894\") " pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:36:22.892383 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.892355 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7xh\" (UniqueName: \"kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh\") pod \"authorino-56c6c9ddd5-86z67\" (UID: \"e4f3dcbf-d30a-4151-9721-78d03fe90894\") " pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:36:22.949445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:22.949378 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7xh\" (UniqueName: \"kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh\") pod \"authorino-56c6c9ddd5-86z67\" (UID: \"e4f3dcbf-d30a-4151-9721-78d03fe90894\") " pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:36:23.014166 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:23.014140 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:36:23.050431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:23.050401 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" event={"ID":"ec367157-0297-4848-8328-d86c512654f5","Type":"ContainerStarted","Data":"a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab"} Nov 03 20:36:23.050790 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:23.050509 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:23.158817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:23.158754 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" podStartSLOduration=2.276618874 podStartE2EDuration="4.158737356s" totalImagesPullingTime="1.882118482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:23.1583531 +0000 UTC m=+391.223594574" watchObservedRunningTime="2026-11-03 20:36:23.158737356 +0000 UTC m=+391.223978830" Nov 03 20:36:23.379438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:23.379399 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-86z67"] Nov 03 20:36:23.385920 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:36:23.385893 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4f3dcbf_d30a_4151_9721_78d03fe90894.slice/crio-66826dcb7a1fcb81ba18f303c8f75f78a33e8b48fd46dc753d8838dec61a8032 WatchSource:0}: Error finding container 66826dcb7a1fcb81ba18f303c8f75f78a33e8b48fd46dc753d8838dec61a8032: Status 404 returned error can't find the container with id 66826dcb7a1fcb81ba18f303c8f75f78a33e8b48fd46dc753d8838dec61a8032 Nov 03 20:36:24.058490 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:24.058442 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" event={"ID":"e4f3dcbf-d30a-4151-9721-78d03fe90894","Type":"ContainerStarted","Data":"66826dcb7a1fcb81ba18f303c8f75f78a33e8b48fd46dc753d8838dec61a8032"} Nov 03 20:36:26.073605 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:26.073564 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4jh69" event={"ID":"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59","Type":"ContainerStarted","Data":"5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b"} Nov 03 20:36:26.181117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:26.181066 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4jh69" podStartSLOduration=2.985154588 podStartE2EDuration="6.181053144s" totalImagesPullingTime="3.195898556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:26.179833368 +0000 UTC m=+394.245074840" watchObservedRunningTime="2026-11-03 20:36:26.181053144 +0000 UTC m=+394.246294617" Nov 03 20:36:27.080527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:27.080488 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" event={"ID":"e4f3dcbf-d30a-4151-9721-78d03fe90894","Type":"ContainerStarted","Data":"6b9fe5f8a40e8203b55a6738b89f873d0d2d014ada4ce5a4dba024ca5d7f304c"} Nov 03 20:36:27.187106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:27.187025 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" podStartSLOduration=2.340902594 podStartE2EDuration="5.187010498s" totalImagesPullingTime="2.846107904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:27.186550237 +0000 UTC m=+395.251791709" watchObservedRunningTime="2026-11-03 20:36:27.187010498 +0000 UTC m=+395.252251999" Nov 03 20:36:27.234875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:27.234838 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4jh69"] Nov 03 20:36:28.085909 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:28.085871 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4jh69" podUID="82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" containerName="authorino" containerID="cri-o://5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b" gracePeriod=30 Nov 03 20:36:28.341387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:28.341328 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:28.442961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:28.442928 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt\" (UniqueName: \"kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt\") pod \"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59\" (UID: \"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59\") " Nov 03 20:36:28.445069 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:28.445034 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt" pod "82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" (UID: "82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59"). InnerVolumeSpecName "kube-api-access-4z9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:28.543926 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:28.543892 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z9kt\" (UniqueName: \"kubernetes.io/projected/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59-kube-api-access-4z9kt\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:29.091454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.091418 5932 generic.go:356] "Generic (PLEG): container finished" podID="82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" containerID="5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b" exitCode=0 Nov 03 20:36:29.091454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.091458 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4jh69" Nov 03 20:36:29.091965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.091471 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4jh69" event={"ID":"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59","Type":"ContainerDied","Data":"5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b"} Nov 03 20:36:29.091965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.091509 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4jh69" event={"ID":"82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59","Type":"ContainerDied","Data":"d8bfe0cd1b698fc632bacad86ed3247c9eec2c9595e4d2fe205150e0c208ba47"} Nov 03 20:36:29.091965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.091531 5932 scope.go:122] "RemoveContainer" containerID="5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b" Nov 03 20:36:29.100083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.100062 5932 scope.go:122] "RemoveContainer" containerID="5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b" Nov 03 20:36:29.100339 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:36:29.100317 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b\": container with ID starting with 5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b not found: ID does not exist" containerID="5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b" Nov 03 20:36:29.100405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.100351 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b"} err="failed to get container status \"5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b\": rpc error: code = NotFound desc = could not find container \"5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b\": container with ID starting with 5b7f56cf528cafac2f0e91f50a4f93f5f2ec75b6c79682de58b519806e038f6b not found: ID does not exist" Nov 03 20:36:29.243868 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.243836 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4jh69"] Nov 03 20:36:29.247985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:29.247963 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4jh69"] Nov 03 20:36:30.049860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.049821 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" path="/var/lib/kubelet/pods/82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59/volumes" Nov 03 20:36:30.175444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.175411 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67"] Nov 03 20:36:30.175854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.175786 5932 cpu_manager.go:414] "containerMap: removing container" podUID="82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" containerName="authorino" Nov 03 20:36:30.175854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.175797 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" containerName="authorino" Nov 03 20:36:30.175922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.175886 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="82942f9f-1d21-4d3f-aaf5-b5c1d93d2c59" containerName="authorino" Nov 03 20:36:30.181374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.181356 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.230579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.230559 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:36:30.256048 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.256021 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.256170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.256058 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jvjx\" (UniqueName: \"kubernetes.io/projected/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-kube-api-access-8jvjx\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.256170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.256149 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.256285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.256264 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.276740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.276718 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:36:30.276962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.276943 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-lg8pr\"" Nov 03 20:36:30.357158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.357087 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.357158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.357129 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jvjx\" (UniqueName: \"kubernetes.io/projected/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-kube-api-access-8jvjx\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.357334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.357161 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.357334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.357246 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.357542 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.357522 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.357604 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.357545 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.369529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.369504 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 20:36:30.371295 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.371274 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67"] Nov 03 20:36:30.378347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.378329 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.414412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.414391 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jvjx\" (UniqueName: \"kubernetes.io/projected/5bd2c162-bfec-4ee1-8b7c-ee3418b07532-kube-api-access-8jvjx\") pod \"kserve-module-controller-manager-7566cf5fc8-9nk67\" (UID: \"5bd2c162-bfec-4ee1-8b7c-ee3418b07532\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.495045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.495022 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:30.912631 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:30.912587 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67"] Nov 03 20:36:30.919723 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:36:30.919690 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd2c162_bfec_4ee1_8b7c_ee3418b07532.slice/crio-d1922387529deef4fa5728bc4c7912f609485451b8606c83c7871b99724ddb36 WatchSource:0}: Error finding container d1922387529deef4fa5728bc4c7912f609485451b8606c83c7871b99724ddb36: Status 404 returned error can't find the container with id d1922387529deef4fa5728bc4c7912f609485451b8606c83c7871b99724ddb36 Nov 03 20:36:31.103210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:31.103179 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" event={"ID":"5bd2c162-bfec-4ee1-8b7c-ee3418b07532","Type":"ContainerStarted","Data":"d1922387529deef4fa5728bc4c7912f609485451b8606c83c7871b99724ddb36"} Nov 03 20:36:34.060248 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:34.060225 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:34.122372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:34.122336 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" event={"ID":"5bd2c162-bfec-4ee1-8b7c-ee3418b07532","Type":"ContainerStarted","Data":"4c68dcef41308fc40688c3f7b5c2b6206bb48d6f40c3d031e6cff0b63121b9cb"} Nov 03 20:36:34.122487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:34.122404 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:34.279005 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:34.278917 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" podStartSLOduration=1.221000561 podStartE2EDuration="4.278903226s" totalImagesPullingTime="3.057902665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:34.278302613 +0000 UTC m=+402.343544087" watchObservedRunningTime="2026-11-03 20:36:34.278903226 +0000 UTC m=+402.344144699" Nov 03 20:36:38.016698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:38.016668 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:38.017066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:38.016874 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" podUID="ec367157-0297-4848-8328-d86c512654f5" containerName="limitador" containerID="cri-o://a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab" gracePeriod=30 Nov 03 20:36:38.953035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:38.953012 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:39.025985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.025958 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9\" (UniqueName: \"kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9\") pod \"ec367157-0297-4848-8328-d86c512654f5\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " Nov 03 20:36:39.026316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.026039 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file\" (UniqueName: \"kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file\") pod \"ec367157-0297-4848-8328-d86c512654f5\" (UID: \"ec367157-0297-4848-8328-d86c512654f5\") " Nov 03 20:36:39.026404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.026384 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file" pod "ec367157-0297-4848-8328-d86c512654f5" (UID: "ec367157-0297-4848-8328-d86c512654f5"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:39.027912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.027887 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9" pod "ec367157-0297-4848-8328-d86c512654f5" (UID: "ec367157-0297-4848-8328-d86c512654f5"). InnerVolumeSpecName "kube-api-access-p6qd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:39.127564 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.127535 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qd9\" (UniqueName: \"kubernetes.io/projected/ec367157-0297-4848-8328-d86c512654f5-kube-api-access-p6qd9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:39.127564 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.127561 5932 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ec367157-0297-4848-8328-d86c512654f5-config-file\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:39.147523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.147491 5932 generic.go:356] "Generic (PLEG): container finished" podID="ec367157-0297-4848-8328-d86c512654f5" containerID="a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab" exitCode=0 Nov 03 20:36:39.147668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.147540 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" Nov 03 20:36:39.147668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.147573 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" event={"ID":"ec367157-0297-4848-8328-d86c512654f5","Type":"ContainerDied","Data":"a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab"} Nov 03 20:36:39.147668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.147638 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vcwsd" event={"ID":"ec367157-0297-4848-8328-d86c512654f5","Type":"ContainerDied","Data":"e6377042a52c8d5e803204742ce6c63fd58fc017b855d76ed7945f6a3e8cd370"} Nov 03 20:36:39.147668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.147660 5932 scope.go:122] "RemoveContainer" containerID="a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab" Nov 03 20:36:39.156592 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.156576 5932 scope.go:122] "RemoveContainer" containerID="a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab" Nov 03 20:36:39.156868 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:36:39.156849 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab\": container with ID starting with a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab not found: ID does not exist" containerID="a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab" Nov 03 20:36:39.156924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.156878 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab"} err="failed to get container status \"a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab\": rpc error: code = NotFound desc = could not find container \"a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab\": container with ID starting with a4294e34dde0a7b52df0470776e90564fb1dcf77298d1bf2aff56beddc34a9ab not found: ID does not exist" Nov 03 20:36:39.303773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.303749 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:39.309316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:39.309293 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vcwsd"] Nov 03 20:36:40.049338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:40.049301 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec367157-0297-4848-8328-d86c512654f5" path="/var/lib/kubelet/pods/ec367157-0297-4848-8328-d86c512654f5/volumes" Nov 03 20:36:42.817427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.817396 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456"] Nov 03 20:36:42.817824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.817810 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ec367157-0297-4848-8328-d86c512654f5" containerName="limitador" Nov 03 20:36:42.817875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.817826 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec367157-0297-4848-8328-d86c512654f5" containerName="limitador" Nov 03 20:36:42.817929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.817893 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec367157-0297-4848-8328-d86c512654f5" containerName="limitador" Nov 03 20:36:42.822211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.822193 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:42.856467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.856433 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zwcz\" (UniqueName: \"kubernetes.io/projected/084ba8dc-5495-4217-86b3-5b253ce4d3cc-kube-api-access-7zwcz\") pod \"postgres-7cdbf6dc44-2h456\" (UID: \"084ba8dc-5495-4217-86b3-5b253ce4d3cc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:42.856590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.856572 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/084ba8dc-5495-4217-86b3-5b253ce4d3cc-data\") pod \"postgres-7cdbf6dc44-2h456\" (UID: \"084ba8dc-5495-4217-86b3-5b253ce4d3cc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:42.872866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.872842 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 20:36:42.873082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.873063 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:36:42.891161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.891137 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456"] Nov 03 20:36:42.918880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.918858 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:36:42.918972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.918882 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rt29p\"" Nov 03 20:36:42.957453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.957423 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/084ba8dc-5495-4217-86b3-5b253ce4d3cc-data\") pod \"postgres-7cdbf6dc44-2h456\" (UID: \"084ba8dc-5495-4217-86b3-5b253ce4d3cc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:42.957599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.957476 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zwcz\" (UniqueName: \"kubernetes.io/projected/084ba8dc-5495-4217-86b3-5b253ce4d3cc-kube-api-access-7zwcz\") pod \"postgres-7cdbf6dc44-2h456\" (UID: \"084ba8dc-5495-4217-86b3-5b253ce4d3cc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:42.957828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:42.957811 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/084ba8dc-5495-4217-86b3-5b253ce4d3cc-data\") pod \"postgres-7cdbf6dc44-2h456\" (UID: \"084ba8dc-5495-4217-86b3-5b253ce4d3cc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:43.022591 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:43.022559 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zwcz\" (UniqueName: \"kubernetes.io/projected/084ba8dc-5495-4217-86b3-5b253ce4d3cc-kube-api-access-7zwcz\") pod \"postgres-7cdbf6dc44-2h456\" (UID: \"084ba8dc-5495-4217-86b3-5b253ce4d3cc\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:43.133951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:43.133928 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:43.502507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:43.502422 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456"] Nov 03 20:36:43.508428 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:36:43.508404 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084ba8dc_5495_4217_86b3_5b253ce4d3cc.slice/crio-8fcde933d8048d3b1ff4f293e7a262323357924d60d6f350fa07ee37e86921c2 WatchSource:0}: Error finding container 8fcde933d8048d3b1ff4f293e7a262323357924d60d6f350fa07ee37e86921c2: Status 404 returned error can't find the container with id 8fcde933d8048d3b1ff4f293e7a262323357924d60d6f350fa07ee37e86921c2 Nov 03 20:36:44.149015 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:36:44.148981 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:36:44.174369 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:44.174339 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" event={"ID":"084ba8dc-5495-4217-86b3-5b253ce4d3cc","Type":"ContainerStarted","Data":"8fcde933d8048d3b1ff4f293e7a262323357924d60d6f350fa07ee37e86921c2"} Nov 03 20:36:51.224187 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:51.224109 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" event={"ID":"084ba8dc-5495-4217-86b3-5b253ce4d3cc","Type":"ContainerStarted","Data":"7123d97a220ceba65e10866527e359b89ae9483cf45dc277c07def744935be6a"} Nov 03 20:36:51.224187 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:51.224171 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:36:51.332910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:51.332836 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" podStartSLOduration=1.8782303009999999 podStartE2EDuration="9.332816204s" totalImagesPullingTime="7.454585903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:36:51.331984894 +0000 UTC m=+419.397226368" watchObservedRunningTime="2026-11-03 20:36:51.332816204 +0000 UTC m=+419.398057678" Nov 03 20:36:55.128717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:55.128685 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-9nk67" Nov 03 20:36:57.258199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:36:57.258173 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2h456" Nov 03 20:37:04.494075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.494038 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wr748"] Nov 03 20:37:04.497340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.497322 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.546777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.546749 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:37:04.546777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.546766 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:37:04.592785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.592752 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fc9dv\"" Nov 03 20:37:04.592785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.592771 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:37:04.594503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.594472 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wr748"] Nov 03 20:37:04.641515 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.641478 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzwh7\" (UniqueName: \"kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.641655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.641520 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.641749 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.641731 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.742640 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.742592 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.742786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.742668 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzwh7\" (UniqueName: \"kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.742786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.742699 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.749486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.746085 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.749486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.746130 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.802207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.802177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzwh7\" (UniqueName: \"kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7\") pod \"maas-controller-579cf7c6d-wr748\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.810993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.810975 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:04.922018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.921983 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-6mgs6"] Nov 03 20:37:04.925762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.925744 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:04.981898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:04.981872 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-6mgs6"] Nov 03 20:37:05.045807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.045749 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.045807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.045777 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vxk\" (UniqueName: \"kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.045959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.045819 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.146943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.146919 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.147064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.146988 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.147064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.147007 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vxk\" (UniqueName: \"kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.149314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.149288 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.149409 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.149316 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.178267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.178241 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wr748"] Nov 03 20:37:05.184412 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:37:05.184388 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3423d7_57dc_41aa_ad7c_d29a0223d831.slice/crio-3d9481b7574d3bb22b372e1cd76c622d305d2b3980a6c47ad4262f15d8142b50 WatchSource:0}: Error finding container 3d9481b7574d3bb22b372e1cd76c622d305d2b3980a6c47ad4262f15d8142b50: Status 404 returned error can't find the container with id 3d9481b7574d3bb22b372e1cd76c622d305d2b3980a6c47ad4262f15d8142b50 Nov 03 20:37:05.204486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.204457 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vxk\" (UniqueName: \"kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk\") pod \"maas-controller-7dcb8dd554-6mgs6\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.238375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.238351 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:05.303915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.303854 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wr748" event={"ID":"2a3423d7-57dc-41aa-ad7c-d29a0223d831","Type":"ContainerStarted","Data":"3d9481b7574d3bb22b372e1cd76c622d305d2b3980a6c47ad4262f15d8142b50"} Nov 03 20:37:05.611643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:05.611551 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-6mgs6"] Nov 03 20:37:05.617661 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:37:05.617635 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc903517_46d5_4392_83e1_a73e1d159bb0.slice/crio-65089972c5fb78cc10c20315264b0ae85bea836d29a6bb28b6c65858ca7e0b28 WatchSource:0}: Error finding container 65089972c5fb78cc10c20315264b0ae85bea836d29a6bb28b6c65858ca7e0b28: Status 404 returned error can't find the container with id 65089972c5fb78cc10c20315264b0ae85bea836d29a6bb28b6c65858ca7e0b28 Nov 03 20:37:06.310759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:06.310724 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" event={"ID":"dc903517-46d5-4392-83e1-a73e1d159bb0","Type":"ContainerStarted","Data":"65089972c5fb78cc10c20315264b0ae85bea836d29a6bb28b6c65858ca7e0b28"} Nov 03 20:37:09.338558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:09.338515 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wr748" event={"ID":"2a3423d7-57dc-41aa-ad7c-d29a0223d831","Type":"ContainerStarted","Data":"bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df"} Nov 03 20:37:09.338998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:09.338729 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:09.340331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:09.340266 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" event={"ID":"dc903517-46d5-4392-83e1-a73e1d159bb0","Type":"ContainerStarted","Data":"03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc"} Nov 03 20:37:09.340331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:09.340321 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:09.467803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:09.467757 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-wr748" podStartSLOduration=1.553435188 podStartE2EDuration="5.467742719s" totalImagesPullingTime="3.914307531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:09.465653452 +0000 UTC m=+437.530894922" watchObservedRunningTime="2026-11-03 20:37:09.467742719 +0000 UTC m=+437.532984191" Nov 03 20:37:09.624667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:09.624596 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" podStartSLOduration=2.08962772 podStartE2EDuration="5.624579998s" totalImagesPullingTime="3.534952278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:09.621108169 +0000 UTC m=+437.686349643" watchObservedRunningTime="2026-11-03 20:37:09.624579998 +0000 UTC m=+437.689821470" Nov 03 20:37:12.343293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.343263 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf"] Nov 03 20:37:12.346959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.346936 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.395583 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.395555 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 20:37:12.395583 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.395568 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 20:37:12.395793 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.395739 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 20:37:12.419498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.419473 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.419649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.419528 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.419649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.419551 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/628e6e07-734c-4a91-81e4-21c61545b56b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.419649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.419609 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5kkn\" (UniqueName: \"kubernetes.io/projected/628e6e07-734c-4a91-81e4-21c61545b56b-kube-api-access-l5kkn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.442213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.442164 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rpvlx\"" Nov 03 20:37:12.445034 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.445003 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf"] Nov 03 20:37:12.520116 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.520075 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.520286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.520128 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.520286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.520164 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/628e6e07-734c-4a91-81e4-21c61545b56b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.520286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.520186 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5kkn\" (UniqueName: \"kubernetes.io/projected/628e6e07-734c-4a91-81e4-21c61545b56b-kube-api-access-l5kkn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.520286 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:12.520246 5932 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 20:37:12.520497 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:12.520307 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-proxy-source podName:628e6e07-734c-4a91-81e4-21c61545b56b nodeName:}" failed. No retries permitted until 2026-11-03 20:37:13.02029158 +0000 UTC m=+441.085533048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" (UID: "628e6e07-734c-4a91-81e4-21c61545b56b") : configmap "usage-tenancy-proxy-source" not found Nov 03 20:37:12.520497 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:12.520325 5932 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 20:37:12.520497 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:12.520380 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628e6e07-734c-4a91-81e4-21c61545b56b-tls-cert podName:628e6e07-734c-4a91-81e4-21c61545b56b nodeName:}" failed. No retries permitted until 2026-11-03 20:37:13.020365182 +0000 UTC m=+441.085606652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/628e6e07-734c-4a91-81e4-21c61545b56b-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" (UID: "628e6e07-734c-4a91-81e4-21c61545b56b") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 20:37:12.520770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.520745 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:12.579182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:12.579156 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5kkn\" (UniqueName: \"kubernetes.io/projected/628e6e07-734c-4a91-81e4-21c61545b56b-kube-api-access-l5kkn\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:13.026253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:13.026216 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/628e6e07-734c-4a91-81e4-21c61545b56b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:13.026445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:13.026322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:13.027086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:13.027060 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/628e6e07-734c-4a91-81e4-21c61545b56b-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:13.028603 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:13.028577 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/628e6e07-734c-4a91-81e4-21c61545b56b-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kqjhf\" (UID: \"628e6e07-734c-4a91-81e4-21c61545b56b\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:13.260977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:13.260941 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:13.629007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:13.628974 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf"] Nov 03 20:37:13.635897 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:37:13.635864 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628e6e07_734c_4a91_81e4_21c61545b56b.slice/crio-97e3f5996f7311de84220471a90469aee418316f74bd8811fb22b6d0ebe65855 WatchSource:0}: Error finding container 97e3f5996f7311de84220471a90469aee418316f74bd8811fb22b6d0ebe65855: Status 404 returned error can't find the container with id 97e3f5996f7311de84220471a90469aee418316f74bd8811fb22b6d0ebe65855 Nov 03 20:37:14.150487 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:14.150442 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:37:14.367544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:14.367510 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" event={"ID":"628e6e07-734c-4a91-81e4-21c61545b56b","Type":"ContainerStarted","Data":"97e3f5996f7311de84220471a90469aee418316f74bd8811fb22b6d0ebe65855"} Nov 03 20:37:20.348835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.348806 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:20.349306 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.348873 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:37:20.517544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.517485 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wr748"] Nov 03 20:37:20.518376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.518340 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-wr748" podUID="2a3423d7-57dc-41aa-ad7c-d29a0223d831" containerName="manager" containerID="cri-o://bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df" gracePeriod=30 Nov 03 20:37:20.783537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.783510 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:20.902006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.901341 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert\") pod \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " Nov 03 20:37:20.902006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.901401 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7\" (UniqueName: \"kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7\") pod \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " Nov 03 20:37:20.902006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.901429 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs\") pod \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\" (UID: \"2a3423d7-57dc-41aa-ad7c-d29a0223d831\") " Nov 03 20:37:20.905384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.905336 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7" pod "2a3423d7-57dc-41aa-ad7c-d29a0223d831" (UID: "2a3423d7-57dc-41aa-ad7c-d29a0223d831"). InnerVolumeSpecName "kube-api-access-dzwh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:37:20.905503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.905458 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert" pod "2a3423d7-57dc-41aa-ad7c-d29a0223d831" (UID: "2a3423d7-57dc-41aa-ad7c-d29a0223d831"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:20.907744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:20.907708 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs" pod "2a3423d7-57dc-41aa-ad7c-d29a0223d831" (UID: "2a3423d7-57dc-41aa-ad7c-d29a0223d831"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:37:21.002584 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.002551 5932 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-webhook-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:21.002584 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.002586 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzwh7\" (UniqueName: \"kubernetes.io/projected/2a3423d7-57dc-41aa-ad7c-d29a0223d831-kube-api-access-dzwh7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:21.002786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.002596 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a3423d7-57dc-41aa-ad7c-d29a0223d831-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:37:21.409417 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.409374 5932 generic.go:356] "Generic (PLEG): container finished" podID="2a3423d7-57dc-41aa-ad7c-d29a0223d831" containerID="bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df" exitCode=0 Nov 03 20:37:21.409914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.409422 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-wr748" Nov 03 20:37:21.409914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.409435 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wr748" event={"ID":"2a3423d7-57dc-41aa-ad7c-d29a0223d831","Type":"ContainerDied","Data":"bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df"} Nov 03 20:37:21.409914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.409486 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-wr748" event={"ID":"2a3423d7-57dc-41aa-ad7c-d29a0223d831","Type":"ContainerDied","Data":"3d9481b7574d3bb22b372e1cd76c622d305d2b3980a6c47ad4262f15d8142b50"} Nov 03 20:37:21.409914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.409515 5932 scope.go:122] "RemoveContainer" containerID="bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df" Nov 03 20:37:21.423410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.423380 5932 scope.go:122] "RemoveContainer" containerID="bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df" Nov 03 20:37:21.423858 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:21.423806 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df\": container with ID starting with bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df not found: ID does not exist" containerID="bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df" Nov 03 20:37:21.423858 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.423839 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df"} err="failed to get container status \"bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df\": rpc error: code = NotFound desc = could not find container \"bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df\": container with ID starting with bfc0d4c2b0164691289b127b6752b7da08450b00115ced86eb2fce9278f900df not found: ID does not exist" Nov 03 20:37:21.563630 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.563587 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wr748"] Nov 03 20:37:21.609924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:21.609891 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-wr748"] Nov 03 20:37:22.050929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:22.050899 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3423d7-57dc-41aa-ad7c-d29a0223d831" path="/var/lib/kubelet/pods/2a3423d7-57dc-41aa-ad7c-d29a0223d831/volumes" Nov 03 20:37:32.476721 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:32.476683 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" event={"ID":"628e6e07-734c-4a91-81e4-21c61545b56b","Type":"ContainerStarted","Data":"0b67609777d9047fa90cbb31fac9873316fee78f01a2acb9398bf14d6e377881"} Nov 03 20:37:32.477156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:32.476744 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:32.594072 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:32.594015 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" podStartSLOduration=2.44660562 podStartE2EDuration="20.594000086s" totalImagesPullingTime="18.147394466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:32.592328539 +0000 UTC m=+460.657570011" watchObservedRunningTime="2026-11-03 20:37:32.594000086 +0000 UTC m=+460.659241590" Nov 03 20:37:33.001174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.001141 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g"] Nov 03 20:37:33.001568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.001556 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2a3423d7-57dc-41aa-ad7c-d29a0223d831" containerName="manager" Nov 03 20:37:33.001630 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.001569 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3423d7-57dc-41aa-ad7c-d29a0223d831" containerName="manager" Nov 03 20:37:33.001686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.001676 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3423d7-57dc-41aa-ad7c-d29a0223d831" containerName="manager" Nov 03 20:37:33.004833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.004814 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.054007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.053980 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:37:33.054007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.053990 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:37:33.054516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.054489 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:37:33.100378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.100356 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dddj5\"" Nov 03 20:37:33.121568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.121542 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g"] Nov 03 20:37:33.124594 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124568 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.124895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124653 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.124895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124742 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.124895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124817 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fmt\" (UniqueName: \"kubernetes.io/projected/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-kube-api-access-r8fmt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.124895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124840 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.124895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124870 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.124895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.124892 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226026 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226072 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fmt\" (UniqueName: \"kubernetes.io/projected/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-kube-api-access-r8fmt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226092 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226127 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226149 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226174 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.226440 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.226306 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.227069 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.227039 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.228536 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.228513 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.228640 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.228557 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.228778 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.228758 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.228818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.228792 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.284880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.284826 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.284975 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.284892 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fmt\" (UniqueName: \"kubernetes.io/projected/f5b16b74-0aeb-4ff9-b150-d9f8b3693d02-kube-api-access-r8fmt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-4zt6g\" (UID: \"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.316836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.316803 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:33.739602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:33.739562 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g"] Nov 03 20:37:33.745867 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:37:33.745837 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b16b74_0aeb_4ff9_b150_d9f8b3693d02.slice/crio-5f61d5db2c31e54459f8492e4c9b9adae87a568fa1e384634d41f27cf5ba8145 WatchSource:0}: Error finding container 5f61d5db2c31e54459f8492e4c9b9adae87a568fa1e384634d41f27cf5ba8145: Status 404 returned error can't find the container with id 5f61d5db2c31e54459f8492e4c9b9adae87a568fa1e384634d41f27cf5ba8145 Nov 03 20:37:34.488662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:34.488597 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" event={"ID":"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02","Type":"ContainerStarted","Data":"5f61d5db2c31e54459f8492e4c9b9adae87a568fa1e384634d41f27cf5ba8145"} Nov 03 20:37:41.074177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:41.074144 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:37:41.074460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:41.074206 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:37:41.526640 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:41.526586 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" event={"ID":"f5b16b74-0aeb-4ff9-b150-d9f8b3693d02","Type":"ContainerStarted","Data":"7b559d1f60ccb649991340f8cdeb23b4578d8a74e83a1b1f6fb15a65cdde9dc5"} Nov 03 20:37:41.526805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:41.526680 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:41.743086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:41.743019 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" podStartSLOduration=2.41759429 podStartE2EDuration="9.743004376s" totalImagesPullingTime="7.325410086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:41.696297806 +0000 UTC m=+469.761539278" watchObservedRunningTime="2026-11-03 20:37:41.743004376 +0000 UTC m=+469.808245848" Nov 03 20:37:42.533689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:42.533653 5932 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-4zt6g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:37:42.534070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:42.533716 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" podUID="f5b16b74-0aeb-4ff9-b150-d9f8b3693d02" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:37:43.486385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:43.486355 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kqjhf" Nov 03 20:37:44.149699 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:37:44.149663 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:37:45.532683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:45.532649 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-4zt6g" Nov 03 20:37:58.620845 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:58.620762 5932 generic.go:356] "Generic (PLEG): container finished" podID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerID="8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89" exitCode=0 Nov 03 20:37:58.621237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:58.620836 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" event={"ID":"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017","Type":"ContainerDied","Data":"8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89"} Nov 03 20:37:58.621237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:58.621183 5932 scope.go:122] "RemoveContainer" containerID="8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89" Nov 03 20:37:58.622271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:58.622241 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:37:58.622339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:58.622316 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:37:58.622375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:58.622350 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:37:59.650776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:59.650743 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" event={"ID":"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017","Type":"ContainerStarted","Data":"9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1"} Nov 03 20:37:59.843646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:59.842930 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:37:59.845777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:59.845751 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:37:59.845927 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:37:59.845802 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 20:38:00.842041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:00.842006 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:00.842670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:00.842614 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 20:38:01.842601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:01.842560 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:01.843250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:01.843212 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 20:38:02.840043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:02.840003 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:02.840316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:02.840281 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 20:38:03.216949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:03.216915 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-86z67"] Nov 03 20:38:03.217332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:03.217096 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" podUID="e4f3dcbf-d30a-4151-9721-78d03fe90894" containerName="authorino" containerID="cri-o://6b9fe5f8a40e8203b55a6738b89f873d0d2d014ada4ce5a4dba024ca5d7f304c" gracePeriod=30 Nov 03 20:38:03.674993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:03.674957 5932 generic.go:356] "Generic (PLEG): container finished" podID="e4f3dcbf-d30a-4151-9721-78d03fe90894" containerID="6b9fe5f8a40e8203b55a6738b89f873d0d2d014ada4ce5a4dba024ca5d7f304c" exitCode=0 Nov 03 20:38:03.675174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:03.675035 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" event={"ID":"e4f3dcbf-d30a-4151-9721-78d03fe90894","Type":"ContainerDied","Data":"6b9fe5f8a40e8203b55a6738b89f873d0d2d014ada4ce5a4dba024ca5d7f304c"} Nov 03 20:38:03.842017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:03.841978 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 20:38:03.842255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:03.842227 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 20:38:04.069067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.069039 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:38:04.156163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.156129 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh\" (UniqueName: \"kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh\") pod \"e4f3dcbf-d30a-4151-9721-78d03fe90894\" (UID: \"e4f3dcbf-d30a-4151-9721-78d03fe90894\") " Nov 03 20:38:04.158163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.158135 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh" pod "e4f3dcbf-d30a-4151-9721-78d03fe90894" (UID: "e4f3dcbf-d30a-4151-9721-78d03fe90894"). InnerVolumeSpecName "kube-api-access-bg7xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:38:04.257469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.257370 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg7xh\" (UniqueName: \"kubernetes.io/projected/e4f3dcbf-d30a-4151-9721-78d03fe90894-kube-api-access-bg7xh\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:38:04.470152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.470117 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:38:04.681755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.681715 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" event={"ID":"e4f3dcbf-d30a-4151-9721-78d03fe90894","Type":"ContainerDied","Data":"66826dcb7a1fcb81ba18f303c8f75f78a33e8b48fd46dc753d8838dec61a8032"} Nov 03 20:38:04.681755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.681734 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-86z67" Nov 03 20:38:04.681965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.681770 5932 scope.go:122] "RemoveContainer" containerID="6b9fe5f8a40e8203b55a6738b89f873d0d2d014ada4ce5a4dba024ca5d7f304c" Nov 03 20:38:04.843255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.843224 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:38:04.844205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.844185 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 20:38:04.845042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.845021 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-86z67"] Nov 03 20:38:04.855011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:04.854991 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-86z67"] Nov 03 20:38:06.049461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:38:06.049427 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f3dcbf-d30a-4151-9721-78d03fe90894" path="/var/lib/kubelet/pods/e4f3dcbf-d30a-4151-9721-78d03fe90894/volumes" Nov 03 20:38:14.149017 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:38:14.148981 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:38:44.149451 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:38:44.149416 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:39:14.148182 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:39:14.148097 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:39:38.175430 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:38.175397 5932 generic.go:356] "Generic (PLEG): container finished" podID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerID="03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc" exitCode=1 Nov 03 20:39:38.175862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:38.175471 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" event={"ID":"dc903517-46d5-4392-83e1-a73e1d159bb0","Type":"ContainerDied","Data":"03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc"} Nov 03 20:39:38.175862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:38.175829 5932 scope.go:122] "RemoveContainer" containerID="03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc" Nov 03 20:39:39.188877 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:39.188841 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" event={"ID":"dc903517-46d5-4392-83e1-a73e1d159bb0","Type":"ContainerStarted","Data":"488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131"} Nov 03 20:39:39.189357 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:39.189100 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:39:44.163593 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:39:44.163554 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:39:50.195881 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:50.195848 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 20:39:52.175948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:52.175920 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:39:52.176914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:52.176893 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:39:54.132003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.131966 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps"] Nov 03 20:39:54.132378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.132366 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e4f3dcbf-d30a-4151-9721-78d03fe90894" containerName="authorino" Nov 03 20:39:54.132378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.132376 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f3dcbf-d30a-4151-9721-78d03fe90894" containerName="authorino" Nov 03 20:39:54.132479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.132468 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f3dcbf-d30a-4151-9721-78d03fe90894" containerName="authorino" Nov 03 20:39:54.135741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.135714 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.220537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.220507 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.220738 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.220562 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.220817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.220738 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.220817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.220781 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.220909 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.220845 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgzvv\" (UniqueName: \"kubernetes.io/projected/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-kube-api-access-mgzvv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.220952 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.220919 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.230732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.230711 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zkx7m\"" Nov 03 20:39:54.230842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.230808 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 20:39:54.232735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.232716 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps"] Nov 03 20:39:54.262254 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.262232 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:39:54.262348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.262256 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:39:54.321757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.321722 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.321938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.321767 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.321938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.321789 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgzvv\" (UniqueName: \"kubernetes.io/projected/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-kube-api-access-mgzvv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.321938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.321885 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.321938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.321926 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.322167 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.322027 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.322451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.322394 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.322451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.322411 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.322451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.322446 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.324659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.324640 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.325052 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.325032 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.384390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.384333 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgzvv\" (UniqueName: \"kubernetes.io/projected/3ab4bbe3-d96e-4e91-9345-d76342e8dc9b-kube-api-access-mgzvv\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps\" (UID: \"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.449005 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.448962 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:39:54.823663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:54.823560 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps"] Nov 03 20:39:54.829156 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:39:54.829131 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab4bbe3_d96e_4e91_9345_d76342e8dc9b.slice/crio-e7e80e38c4114bbe4dccb0e4a3abf7abdbe09eac82308474d345b734b760d42d WatchSource:0}: Error finding container e7e80e38c4114bbe4dccb0e4a3abf7abdbe09eac82308474d345b734b760d42d: Status 404 returned error can't find the container with id e7e80e38c4114bbe4dccb0e4a3abf7abdbe09eac82308474d345b734b760d42d Nov 03 20:39:55.277697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:55.277656 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" event={"ID":"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b","Type":"ContainerStarted","Data":"e7e80e38c4114bbe4dccb0e4a3abf7abdbe09eac82308474d345b734b760d42d"} Nov 03 20:39:57.291513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:57.291478 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" event={"ID":"3ab4bbe3-d96e-4e91-9345-d76342e8dc9b","Type":"ContainerStarted","Data":"bbd06d7b0c3f059a8b595c2e4a05c7ea0f7a6eb2dbd387b18de372f65c22e03c"} Nov 03 20:39:57.403840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:57.403786 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" podStartSLOduration=1.839685199 podStartE2EDuration="3.403770659s" totalImagesPullingTime="1.56408546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:39:57.401942678 +0000 UTC m=+605.467184153" watchObservedRunningTime="2026-11-03 20:39:57.403770659 +0000 UTC m=+605.469012153" Nov 03 20:39:57.449484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:39:57.449455 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:40:04.385341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.385303 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj"] Nov 03 20:40:04.389151 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.389131 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.416259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.416229 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.416392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.416275 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.416392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.416373 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx69c\" (UniqueName: \"kubernetes.io/projected/87927855-684d-4f9a-8614-dd9649b8af0b-kube-api-access-hx69c\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.416507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.416424 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.416567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.416524 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.416674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.416657 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87927855-684d-4f9a-8614-dd9649b8af0b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.449817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.449792 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:40:04.462308 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.462287 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:40:04.486925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.486901 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 20:40:04.488761 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.488740 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj"] Nov 03 20:40:04.517668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.517645 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx69c\" (UniqueName: \"kubernetes.io/projected/87927855-684d-4f9a-8614-dd9649b8af0b-kube-api-access-hx69c\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.517759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.517688 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.517759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.517714 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.517875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.517772 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87927855-684d-4f9a-8614-dd9649b8af0b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.517875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.517799 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.517875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.517825 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.518289 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.518258 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.518384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.518292 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.518384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.518342 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.519909 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.519882 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87927855-684d-4f9a-8614-dd9649b8af0b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.520203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.520185 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87927855-684d-4f9a-8614-dd9649b8af0b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.574882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.574844 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx69c\" (UniqueName: \"kubernetes.io/projected/87927855-684d-4f9a-8614-dd9649b8af0b-kube-api-access-hx69c\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-5vmsj\" (UID: \"87927855-684d-4f9a-8614-dd9649b8af0b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:04.701820 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:04.701743 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:05.074949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:05.074860 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj"] Nov 03 20:40:05.081100 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:40:05.081069 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87927855_684d_4f9a_8614_dd9649b8af0b.slice/crio-66d59c68ad0c8c3da7342c597b8b3f6dc038012cb80bdf0c809767ada8cb2ccd WatchSource:0}: Error finding container 66d59c68ad0c8c3da7342c597b8b3f6dc038012cb80bdf0c809767ada8cb2ccd: Status 404 returned error can't find the container with id 66d59c68ad0c8c3da7342c597b8b3f6dc038012cb80bdf0c809767ada8cb2ccd Nov 03 20:40:05.082440 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:05.082422 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:40:05.338305 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:05.338223 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" event={"ID":"87927855-684d-4f9a-8614-dd9649b8af0b","Type":"ContainerStarted","Data":"66d59c68ad0c8c3da7342c597b8b3f6dc038012cb80bdf0c809767ada8cb2ccd"} Nov 03 20:40:05.349701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:05.349678 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" Nov 03 20:40:06.345554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:06.345519 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" event={"ID":"87927855-684d-4f9a-8614-dd9649b8af0b","Type":"ContainerStarted","Data":"dc196242e97420687658523405e19716227cde6cc7a417defd091d72f12927d8"} Nov 03 20:40:06.457825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:06.457777 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" podStartSLOduration=1.854467852 podStartE2EDuration="2.457763543s" totalImagesPullingTime="603.295691ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:06.456117725 +0000 UTC m=+614.521359198" watchObservedRunningTime="2026-11-03 20:40:06.457763543 +0000 UTC m=+614.523005007" Nov 03 20:40:06.702223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:06.702195 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:13.200945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.200910 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779"] Nov 03 20:40:13.204762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.204741 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.284364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.284335 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 20:40:13.299977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.299942 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476286d-e752-4bdc-9441-5518c4860345-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.300148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.300001 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.300148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.300069 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2c7l\" (UniqueName: \"kubernetes.io/projected/b476286d-e752-4bdc-9441-5518c4860345-kube-api-access-b2c7l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.300266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.300207 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.300355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.300337 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.300421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.300406 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.303068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.303039 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779"] Nov 03 20:40:13.401464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.401427 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.401464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.401464 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476286d-e752-4bdc-9441-5518c4860345-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.401755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.401488 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.401755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.401506 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2c7l\" (UniqueName: \"kubernetes.io/projected/b476286d-e752-4bdc-9441-5518c4860345-kube-api-access-b2c7l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.401755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.401561 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.401755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.401649 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.402045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.402021 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.402111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.402072 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.402111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.402085 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.403848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.403823 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b476286d-e752-4bdc-9441-5518c4860345-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.404013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.403997 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b476286d-e752-4bdc-9441-5518c4860345-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.459308 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.459242 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2c7l\" (UniqueName: \"kubernetes.io/projected/b476286d-e752-4bdc-9441-5518c4860345-kube-api-access-b2c7l\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-qr779\" (UID: \"b476286d-e752-4bdc-9441-5518c4860345\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.517867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.517839 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:13.901867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:13.901838 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779"] Nov 03 20:40:13.908384 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:40:13.908356 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb476286d_e752_4bdc_9441_5518c4860345.slice/crio-b5d7c6f7df7fb4b17859d85e2b75f63938358049504baf7eba90883667aa0ed2 WatchSource:0}: Error finding container b5d7c6f7df7fb4b17859d85e2b75f63938358049504baf7eba90883667aa0ed2: Status 404 returned error can't find the container with id b5d7c6f7df7fb4b17859d85e2b75f63938358049504baf7eba90883667aa0ed2 Nov 03 20:40:14.155272 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:40:14.155188 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:40:14.391020 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:14.390985 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" event={"ID":"b476286d-e752-4bdc-9441-5518c4860345","Type":"ContainerStarted","Data":"b5d7c6f7df7fb4b17859d85e2b75f63938358049504baf7eba90883667aa0ed2"} Nov 03 20:40:14.702583 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:14.702546 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:14.715190 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:14.715167 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:15.398932 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:15.398895 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" event={"ID":"b476286d-e752-4bdc-9441-5518c4860345","Type":"ContainerStarted","Data":"841140f355d1e32142eabe246b09f6380418fb13f8b31c8427d92a45f9af5cb6"} Nov 03 20:40:15.410051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:15.410026 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" Nov 03 20:40:15.510475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:15.510425 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" podStartSLOduration=1.918483842 podStartE2EDuration="2.510411709s" totalImagesPullingTime="591.927867ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:15.508842437 +0000 UTC m=+623.574083911" watchObservedRunningTime="2026-11-03 20:40:15.510411709 +0000 UTC m=+623.575653182" Nov 03 20:40:15.518775 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:15.518746 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:23.518278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:23.518249 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:23.530682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:23.530659 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:24.458798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:24.458768 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" Nov 03 20:40:30.008436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.008396 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q"] Nov 03 20:40:30.012339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.012316 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.110195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.110144 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-q9pxf\"" Nov 03 20:40:30.110726 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.110527 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 20:40:30.110726 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.110678 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 20:40:30.113793 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.113764 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q"] Nov 03 20:40:30.162160 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.162130 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q298h\" (UniqueName: \"kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h\") pod \"ai-gateway-controller-7cb8f6bb64-j7g7q\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.162312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.162185 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-j7g7q\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.263759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.263689 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q298h\" (UniqueName: \"kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h\") pod \"ai-gateway-controller-7cb8f6bb64-j7g7q\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.263759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.263740 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-j7g7q\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.266157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.266132 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-j7g7q\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.321539 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.321512 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q298h\" (UniqueName: \"kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h\") pod \"ai-gateway-controller-7cb8f6bb64-j7g7q\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.324077 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.324059 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:30.699885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:30.699857 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q"] Nov 03 20:40:30.706431 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:40:30.706395 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf548a685_bbdc_40da_9175_5a929469e0ee.slice/crio-e414f2e3e2d4247f5a910a6841217b1f93165f0d8713ea35ca50990fce0c91c4 WatchSource:0}: Error finding container e414f2e3e2d4247f5a910a6841217b1f93165f0d8713ea35ca50990fce0c91c4: Status 404 returned error can't find the container with id e414f2e3e2d4247f5a910a6841217b1f93165f0d8713ea35ca50990fce0c91c4 Nov 03 20:40:31.484109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:31.484075 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" event={"ID":"f548a685-bbdc-40da-9175-5a929469e0ee","Type":"ContainerStarted","Data":"e414f2e3e2d4247f5a910a6841217b1f93165f0d8713ea35ca50990fce0c91c4"} Nov 03 20:40:33.498503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:33.498460 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" event={"ID":"f548a685-bbdc-40da-9175-5a929469e0ee","Type":"ContainerStarted","Data":"5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3"} Nov 03 20:40:33.498925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:33.498540 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:33.606960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:33.606897 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" podStartSLOduration=2.608308567 podStartE2EDuration="4.606827958s" totalImagesPullingTime="1.998519391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:33.60600417 +0000 UTC m=+641.671245642" watchObservedRunningTime="2026-11-03 20:40:33.606827958 +0000 UTC m=+641.672069435" Nov 03 20:40:37.156538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.156500 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-mdntw"] Nov 03 20:40:37.159058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.159040 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.207890 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.207855 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 20:40:37.254046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.254013 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nmvw4\"" Nov 03 20:40:37.256398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.256367 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-mdntw"] Nov 03 20:40:37.327089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.327063 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e655053f-68d6-4b7e-a193-49de67eb9f01-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-mdntw\" (UID: \"e655053f-68d6-4b7e-a193-49de67eb9f01\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.327227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.327105 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsjrk\" (UniqueName: \"kubernetes.io/projected/e655053f-68d6-4b7e-a193-49de67eb9f01-kube-api-access-dsjrk\") pod \"payload-processing-55fbbbf9dc-mdntw\" (UID: \"e655053f-68d6-4b7e-a193-49de67eb9f01\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.427694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.427596 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e655053f-68d6-4b7e-a193-49de67eb9f01-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-mdntw\" (UID: \"e655053f-68d6-4b7e-a193-49de67eb9f01\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.427694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.427684 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsjrk\" (UniqueName: \"kubernetes.io/projected/e655053f-68d6-4b7e-a193-49de67eb9f01-kube-api-access-dsjrk\") pod \"payload-processing-55fbbbf9dc-mdntw\" (UID: \"e655053f-68d6-4b7e-a193-49de67eb9f01\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.428263 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.428239 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e655053f-68d6-4b7e-a193-49de67eb9f01-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-mdntw\" (UID: \"e655053f-68d6-4b7e-a193-49de67eb9f01\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.485390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.485346 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsjrk\" (UniqueName: \"kubernetes.io/projected/e655053f-68d6-4b7e-a193-49de67eb9f01-kube-api-access-dsjrk\") pod \"payload-processing-55fbbbf9dc-mdntw\" (UID: \"e655053f-68d6-4b7e-a193-49de67eb9f01\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:37.771447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:37.771380 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:38.142554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:38.142525 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-mdntw"] Nov 03 20:40:38.151768 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:40:38.151743 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode655053f_68d6_4b7e_a193_49de67eb9f01.slice/crio-565b5c2ce57da0a2b378cd294c68e667f0e321b28d3eca469f15f8b1ea36a215 WatchSource:0}: Error finding container 565b5c2ce57da0a2b378cd294c68e667f0e321b28d3eca469f15f8b1ea36a215: Status 404 returned error can't find the container with id 565b5c2ce57da0a2b378cd294c68e667f0e321b28d3eca469f15f8b1ea36a215 Nov 03 20:40:38.525472 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:38.525382 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" event={"ID":"e655053f-68d6-4b7e-a193-49de67eb9f01","Type":"ContainerStarted","Data":"565b5c2ce57da0a2b378cd294c68e667f0e321b28d3eca469f15f8b1ea36a215"} Nov 03 20:40:41.545341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:41.545305 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" event={"ID":"e655053f-68d6-4b7e-a193-49de67eb9f01","Type":"ContainerStarted","Data":"6bc2c25c5cbfeb666c7a1fb672d22c23541d6dfd1c4efd4d844e543aa77faf50"} Nov 03 20:40:41.545846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:41.545437 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:41.654781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:41.654733 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" podStartSLOduration=2.129468193 podStartE2EDuration="4.654719646s" totalImagesPullingTime="2.525251453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:40:41.653022542 +0000 UTC m=+649.718264016" watchObservedRunningTime="2026-11-03 20:40:41.654719646 +0000 UTC m=+649.719961120" Nov 03 20:40:44.150165 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:40:44.150126 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:40:44.506264 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:44.506176 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:52.554836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:52.554805 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-mdntw" Nov 03 20:40:58.912915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:58.912881 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q"] Nov 03 20:40:58.913519 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:58.913188 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" podUID="f548a685-bbdc-40da-9175-5a929469e0ee" containerName="manager" containerID="cri-o://5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3" gracePeriod=30 Nov 03 20:40:59.162028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.162004 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:59.232827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.232772 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs\") pod \"f548a685-bbdc-40da-9175-5a929469e0ee\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " Nov 03 20:40:59.232827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.232816 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h\" (UniqueName: \"kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h\") pod \"f548a685-bbdc-40da-9175-5a929469e0ee\" (UID: \"f548a685-bbdc-40da-9175-5a929469e0ee\") " Nov 03 20:40:59.234764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.234738 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs" pod "f548a685-bbdc-40da-9175-5a929469e0ee" (UID: "f548a685-bbdc-40da-9175-5a929469e0ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:59.234861 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.234808 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h" pod "f548a685-bbdc-40da-9175-5a929469e0ee" (UID: "f548a685-bbdc-40da-9175-5a929469e0ee"). InnerVolumeSpecName "kube-api-access-q298h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:59.333966 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.333935 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f548a685-bbdc-40da-9175-5a929469e0ee-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:59.333966 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.333964 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q298h\" (UniqueName: \"kubernetes.io/projected/f548a685-bbdc-40da-9175-5a929469e0ee-kube-api-access-q298h\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:59.642946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.642915 5932 generic.go:356] "Generic (PLEG): container finished" podID="f548a685-bbdc-40da-9175-5a929469e0ee" containerID="5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3" exitCode=0 Nov 03 20:40:59.643136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.642962 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" Nov 03 20:40:59.643136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.643007 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" event={"ID":"f548a685-bbdc-40da-9175-5a929469e0ee","Type":"ContainerDied","Data":"5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3"} Nov 03 20:40:59.643136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.643065 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q" event={"ID":"f548a685-bbdc-40da-9175-5a929469e0ee","Type":"ContainerDied","Data":"e414f2e3e2d4247f5a910a6841217b1f93165f0d8713ea35ca50990fce0c91c4"} Nov 03 20:40:59.643136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.643096 5932 scope.go:122] "RemoveContainer" containerID="5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3" Nov 03 20:40:59.652919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.652903 5932 scope.go:122] "RemoveContainer" containerID="5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3" Nov 03 20:40:59.653193 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:40:59.653174 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3\": container with ID starting with 5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3 not found: ID does not exist" containerID="5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3" Nov 03 20:40:59.653261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.653201 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3"} err="failed to get container status \"5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3\": rpc error: code = NotFound desc = could not find container \"5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3\": container with ID starting with 5df169c3eb8ab5c81085264abae0f6ef9f16b64848d0c61413b4559fc221dce3 not found: ID does not exist" Nov 03 20:40:59.796243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.796216 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q"] Nov 03 20:40:59.797989 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:40:59.797969 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-j7g7q"] Nov 03 20:41:00.050425 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:00.050333 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f548a685-bbdc-40da-9175-5a929469e0ee" path="/var/lib/kubelet/pods/f548a685-bbdc-40da-9175-5a929469e0ee/volumes" Nov 03 20:41:14.149473 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:41:14.149436 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:41:44.149878 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:41:44.149833 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:41:51.769112 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.769071 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx"] Nov 03 20:41:51.769587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.769536 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f548a685-bbdc-40da-9175-5a929469e0ee" containerName="manager" Nov 03 20:41:51.769587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.769547 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f548a685-bbdc-40da-9175-5a929469e0ee" containerName="manager" Nov 03 20:41:51.769712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.769652 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f548a685-bbdc-40da-9175-5a929469e0ee" containerName="manager" Nov 03 20:41:51.772999 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.772981 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.822155 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.822120 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3b8feb-openshift-default-dockercfg-q9mnd\"" Nov 03 20:41:51.870149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.870116 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx"] Nov 03 20:41:51.915752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.915710 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f28k\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.915752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.915753 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916005 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.915780 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916005 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.915978 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.916018 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.916056 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.916139 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.916197 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:51.916260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:51.916242 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.016795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016763 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.016795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016798 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016851 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f28k\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016869 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016888 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016921 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016945 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.016980 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.017010 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.017252 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.017362 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017539 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.017516 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.017599 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.017884 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.017863 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.019396 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.019335 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.019587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.019568 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.080151 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.080121 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.085429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.085402 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f28k\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.090919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.090898 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn"] Nov 03 20:41:52.095929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.095909 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.133261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.133237 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3b8feb-openshift-default-dockercfg-q9mnd\"" Nov 03 20:41:52.136301 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.136272 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:52.144701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.144676 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3d2008-openshift-default-dockercfg-z445t\"" Nov 03 20:41:52.159302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.158756 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn"] Nov 03 20:41:52.218967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.218937 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.218983 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219095 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219152 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219185 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219212 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219261 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219323 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.219408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.219358 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrznl\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320087 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320126 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320144 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320180 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320218 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320239 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrznl\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320271 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320307 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320356 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320586 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320646 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320694 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.320787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.320731 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.321080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.321056 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.322775 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.322753 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.323040 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.323016 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.389191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.389159 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrznl\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.390200 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.390171 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.409233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.409206 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:52.568442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.568408 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx"] Nov 03 20:41:52.574049 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:41:52.573969 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4826cc8_9dd1_4e79_a2ba_8df64125e818.slice/crio-59bb2ff98340e0ea65b0504c09a23cf98eacaecd30ff0d31cc0a55b2853b3978 WatchSource:0}: Error finding container 59bb2ff98340e0ea65b0504c09a23cf98eacaecd30ff0d31cc0a55b2853b3978: Status 404 returned error can't find the container with id 59bb2ff98340e0ea65b0504c09a23cf98eacaecd30ff0d31cc0a55b2853b3978 Nov 03 20:41:52.576483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.576428 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:52.576585 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.576514 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:52.576585 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.576560 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:52.889127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.889083 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn"] Nov 03 20:41:52.902117 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:41:52.902084 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051ea739_77c3_43ae_bcda_7d1915deeb20.slice/crio-3aa9a33f290cea07db93e9f5969212cc44eb1359fe0bf1e844f7293ce50b15fd WatchSource:0}: Error finding container 3aa9a33f290cea07db93e9f5969212cc44eb1359fe0bf1e844f7293ce50b15fd: Status 404 returned error can't find the container with id 3aa9a33f290cea07db93e9f5969212cc44eb1359fe0bf1e844f7293ce50b15fd Nov 03 20:41:52.904891 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.904846 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:52.904993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.904944 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:52.904993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.904986 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:52.942339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.942300 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" event={"ID":"051ea739-77c3-43ae-bcda-7d1915deeb20","Type":"ContainerStarted","Data":"3aa9a33f290cea07db93e9f5969212cc44eb1359fe0bf1e844f7293ce50b15fd"} Nov 03 20:41:52.945178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.944257 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" event={"ID":"c4826cc8-9dd1-4e79-a2ba-8df64125e818","Type":"ContainerStarted","Data":"1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892"} Nov 03 20:41:52.945178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:52.944309 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" event={"ID":"c4826cc8-9dd1-4e79-a2ba-8df64125e818","Type":"ContainerStarted","Data":"59bb2ff98340e0ea65b0504c09a23cf98eacaecd30ff0d31cc0a55b2853b3978"} Nov 03 20:41:53.083057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.083006 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podStartSLOduration=2.082990881 podStartE2EDuration="2.082990881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:53.082726858 +0000 UTC m=+721.147968332" watchObservedRunningTime="2026-11-03 20:41:53.082990881 +0000 UTC m=+721.148232353" Nov 03 20:41:53.124980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.124943 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h"] Nov 03 20:41:53.130091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.130062 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.136446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.136414 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:53.144318 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.144289 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:53.182466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.182434 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0cd49b-openshift-default-dockercfg-qtdr4\"" Nov 03 20:41:53.188194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.188163 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h"] Nov 03 20:41:53.234287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234251 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234314 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234434 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qz2\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234571 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234491 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234609 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234598 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234631 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234693 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234749 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234713 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.234819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.234795 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.335754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.335718 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qz2\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.335950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.335779 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.335950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.335815 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.335950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.335844 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.335981 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336026 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336083 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336217 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336257 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336268 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336520 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336669 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336612 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336724 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.336895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.336870 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.338374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.338345 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.338648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.338606 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.412355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.400413 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.412355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.400725 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qz2\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.450082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.450052 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:53.832035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.831951 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h"] Nov 03 20:41:53.838048 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:41:53.838022 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9a630a_9407_40e4_b964_61b39dd48e75.slice/crio-ea9258e14fc17a24572076c7457d67e2730673306a03866ef13d540ed8802a5f WatchSource:0}: Error finding container ea9258e14fc17a24572076c7457d67e2730673306a03866ef13d540ed8802a5f: Status 404 returned error can't find the container with id ea9258e14fc17a24572076c7457d67e2730673306a03866ef13d540ed8802a5f Nov 03 20:41:53.840444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.840413 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:53.840519 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.840478 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:53.840519 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.840505 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:41:53.956710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.952502 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" event={"ID":"051ea739-77c3-43ae-bcda-7d1915deeb20","Type":"ContainerStarted","Data":"0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e"} Nov 03 20:41:53.956710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.954431 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" event={"ID":"6f9a630a-9407-40e4-b964-61b39dd48e75","Type":"ContainerStarted","Data":"60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03"} Nov 03 20:41:53.956710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.954469 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" event={"ID":"6f9a630a-9407-40e4-b964-61b39dd48e75","Type":"ContainerStarted","Data":"ea9258e14fc17a24572076c7457d67e2730673306a03866ef13d540ed8802a5f"} Nov 03 20:41:53.956710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.954520 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:53.956710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:53.955729 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:41:54.067633 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.067553 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podStartSLOduration=2.067537568 podStartE2EDuration="2.067537568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:54.065597684 +0000 UTC m=+722.130839155" watchObservedRunningTime="2026-11-03 20:41:54.067537568 +0000 UTC m=+722.132779041" Nov 03 20:41:54.301268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.301212 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podStartSLOduration=1.301193844 podStartE2EDuration="1.301193844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:41:54.29720139 +0000 UTC m=+722.362442891" watchObservedRunningTime="2026-11-03 20:41:54.301193844 +0000 UTC m=+722.366435316" Nov 03 20:41:54.410653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.410604 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:54.416297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.416269 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:54.450702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.450669 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:54.455910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.455887 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:54.960086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.960054 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:54.960553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.960106 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:41:54.960716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.960698 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:41:54.960818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:41:54.960744 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:42:04.699213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.699138 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj"] Nov 03 20:42:04.704365 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.704340 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.756510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.756483 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj"] Nov 03 20:42:04.773752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.773711 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlcgb\" (UniqueName: \"kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb\") pod \"payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.774209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.774175 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.805920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.805877 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3b8feb\"" Nov 03 20:42:04.806163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.805991 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3b8feb-dockercfg-v2kkm\"" Nov 03 20:42:04.848506 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.848478 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69"] Nov 03 20:42:04.852233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.852215 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:04.875054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.875025 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:04.875165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.875062 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfxz\" (UniqueName: \"kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz\") pod \"payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:04.875165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.875093 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlcgb\" (UniqueName: \"kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb\") pod \"payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.875285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.875264 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.875991 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.875972 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.910807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.910780 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69"] Nov 03 20:42:04.932680 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.932647 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlcgb\" (UniqueName: \"kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb\") pod \"payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:04.976834 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.976776 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:04.976834 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.976805 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfxz\" (UniqueName: \"kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz\") pod \"payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:04.977415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:04.977396 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:05.018242 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:05.018219 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:05.034798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:05.034771 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfxz\" (UniqueName: \"kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz\") pod \"payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:05.165824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:05.165787 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:05.387194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:05.387155 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj"] Nov 03 20:42:05.393512 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:05.393484 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa65f1a0_75cc_4dfa_97b8_51c8a3089012.slice/crio-b37fd6f25577cb6703004d6dddee39447708b770df8955734dcd2a7da4fc74d8 WatchSource:0}: Error finding container b37fd6f25577cb6703004d6dddee39447708b770df8955734dcd2a7da4fc74d8: Status 404 returned error can't find the container with id b37fd6f25577cb6703004d6dddee39447708b770df8955734dcd2a7da4fc74d8 Nov 03 20:42:05.537276 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:05.537248 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69"] Nov 03 20:42:05.543170 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:05.543141 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode946cf50_c4f9_48ec_b22f_3046232b4601.slice/crio-54bc73089db535a572c76f42eb54ea5f3c88438a060bef6a3aab97c822895d4e WatchSource:0}: Error finding container 54bc73089db535a572c76f42eb54ea5f3c88438a060bef6a3aab97c822895d4e: Status 404 returned error can't find the container with id 54bc73089db535a572c76f42eb54ea5f3c88438a060bef6a3aab97c822895d4e Nov 03 20:42:06.026032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.025985 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" event={"ID":"fa65f1a0-75cc-4dfa-97b8-51c8a3089012","Type":"ContainerStarted","Data":"f28ef7dc96aa9d80ae8c908ec58d0ca965282d8f783402d1f2fa15cd59c5147a"} Nov 03 20:42:06.026663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.026610 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:06.026780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.026676 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" event={"ID":"fa65f1a0-75cc-4dfa-97b8-51c8a3089012","Type":"ContainerStarted","Data":"b37fd6f25577cb6703004d6dddee39447708b770df8955734dcd2a7da4fc74d8"} Nov 03 20:42:06.028108 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.028081 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" event={"ID":"e946cf50-c4f9-48ec-b22f-3046232b4601","Type":"ContainerStarted","Data":"91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3"} Nov 03 20:42:06.028231 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.028132 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" event={"ID":"e946cf50-c4f9-48ec-b22f-3046232b4601","Type":"ContainerStarted","Data":"54bc73089db535a572c76f42eb54ea5f3c88438a060bef6a3aab97c822895d4e"} Nov 03 20:42:06.028231 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.028179 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:06.134038 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.133971 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" podStartSLOduration=2.1339555629999998 podStartE2EDuration="2.133955563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:06.133296895 +0000 UTC m=+734.198538372" watchObservedRunningTime="2026-11-03 20:42:06.133955563 +0000 UTC m=+734.199197036" Nov 03 20:42:06.287195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:06.287064 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" podStartSLOduration=2.287041736 podStartE2EDuration="2.287041736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:06.286222901 +0000 UTC m=+734.351464374" watchObservedRunningTime="2026-11-03 20:42:06.287041736 +0000 UTC m=+734.352283211" Nov 03 20:42:09.684685 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.684651 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl"] Nov 03 20:42:09.688345 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.688325 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.725781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.725756 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssbj7\" (UniqueName: \"kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7\") pod \"payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.725892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.725795 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.742484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.742460 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl"] Nov 03 20:42:09.785753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.785697 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-88228a\"" Nov 03 20:42:09.785907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.785772 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-88228a-dockercfg-xlpz7\"" Nov 03 20:42:09.826998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.826959 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.827179 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.827124 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssbj7\" (UniqueName: \"kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7\") pod \"payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.828169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.828143 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.847129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.847099 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn"] Nov 03 20:42:09.852049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.852022 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:09.887183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.887152 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn"] Nov 03 20:42:09.887679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.887255 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssbj7\" (UniqueName: \"kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7\") pod \"payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:09.927912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.927887 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:09.928041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:09.927965 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5lb\" (UniqueName: \"kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb\") pod \"payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:10.001829 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.001758 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:10.029165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.029135 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:10.029266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.029237 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5lb\" (UniqueName: \"kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb\") pod \"payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:10.029907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.029887 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:10.087895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.087865 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5lb\" (UniqueName: \"kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb\") pod \"payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:10.183646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.183588 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:10.375326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.375291 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl"] Nov 03 20:42:10.381032 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:10.381005 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda026c0ac_78fd_4adf_be17_ef0b5a98619f.slice/crio-64e4486dd5ba5bd74c78bd6d04492070485e8a438aba83052f948c0fd4e14acc WatchSource:0}: Error finding container 64e4486dd5ba5bd74c78bd6d04492070485e8a438aba83052f948c0fd4e14acc: Status 404 returned error can't find the container with id 64e4486dd5ba5bd74c78bd6d04492070485e8a438aba83052f948c0fd4e14acc Nov 03 20:42:10.574236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:10.574202 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn"] Nov 03 20:42:10.577170 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:10.577132 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5193f5f2_13c8_4ea5_aaa3_618fa746beae.slice/crio-eb09e3518d02c44885b64ef0154b327a4a794bf05fff4cd6f4920cad143274bb WatchSource:0}: Error finding container eb09e3518d02c44885b64ef0154b327a4a794bf05fff4cd6f4920cad143274bb: Status 404 returned error can't find the container with id eb09e3518d02c44885b64ef0154b327a4a794bf05fff4cd6f4920cad143274bb Nov 03 20:42:11.064691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.064576 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" event={"ID":"5193f5f2-13c8-4ea5-aaa3-618fa746beae","Type":"ContainerStarted","Data":"d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73"} Nov 03 20:42:11.064691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.064663 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:11.064691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.064690 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" event={"ID":"5193f5f2-13c8-4ea5-aaa3-618fa746beae","Type":"ContainerStarted","Data":"eb09e3518d02c44885b64ef0154b327a4a794bf05fff4cd6f4920cad143274bb"} Nov 03 20:42:11.066196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.066141 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" event={"ID":"a026c0ac-78fd-4adf-be17-ef0b5a98619f","Type":"ContainerStarted","Data":"638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1"} Nov 03 20:42:11.066286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.066219 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:11.066286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.066241 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" event={"ID":"a026c0ac-78fd-4adf-be17-ef0b5a98619f","Type":"ContainerStarted","Data":"64e4486dd5ba5bd74c78bd6d04492070485e8a438aba83052f948c0fd4e14acc"} Nov 03 20:42:11.173682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.173597 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" podStartSLOduration=2.17358003 podStartE2EDuration="2.17358003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:11.172932046 +0000 UTC m=+739.238173519" watchObservedRunningTime="2026-11-03 20:42:11.17358003 +0000 UTC m=+739.238821494" Nov 03 20:42:11.297098 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:11.295931 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" podStartSLOduration=2.29590753 podStartE2EDuration="2.29590753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:11.291708647 +0000 UTC m=+739.356950122" watchObservedRunningTime="2026-11-03 20:42:11.29590753 +0000 UTC m=+739.361149003" Nov 03 20:42:12.250820 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.250786 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw"] Nov 03 20:42:12.255986 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.255961 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.369693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.368933 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3d2008-dockercfg-nr5wp\"" Nov 03 20:42:12.369693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.369060 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3d2008\"" Nov 03 20:42:12.392940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.392517 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb46c\" (UniqueName: \"kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c\") pod \"payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.392940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.392677 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.393218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.393151 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw"] Nov 03 20:42:12.416041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.413222 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v"] Nov 03 20:42:12.418121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.417880 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.461657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.453966 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v"] Nov 03 20:42:12.493456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.493424 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.493589 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.493563 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb46c\" (UniqueName: \"kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c\") pod \"payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.493636 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.493595 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzntb\" (UniqueName: \"kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb\") pod \"payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.493680 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.493662 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.494292 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.494273 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.552154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.551743 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb46c\" (UniqueName: \"kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c\") pod \"payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.570208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.570181 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:12.595217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.595170 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzntb\" (UniqueName: \"kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb\") pod \"payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.595316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.595292 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.596151 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.596128 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.657262 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.657237 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzntb\" (UniqueName: \"kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb\") pod \"payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.769414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.769381 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:12.968014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:12.967919 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw"] Nov 03 20:42:12.974047 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:12.974019 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3647f3e8_3643_4845_b8ac_b7577b04ba1a.slice/crio-6990ca051ca0da46fcacaede7998d89357fc0be7e1f749afa432063bcd11bfa1 WatchSource:0}: Error finding container 6990ca051ca0da46fcacaede7998d89357fc0be7e1f749afa432063bcd11bfa1: Status 404 returned error can't find the container with id 6990ca051ca0da46fcacaede7998d89357fc0be7e1f749afa432063bcd11bfa1 Nov 03 20:42:13.078818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:13.078789 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" event={"ID":"3647f3e8-3643-4845-b8ac-b7577b04ba1a","Type":"ContainerStarted","Data":"248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a"} Nov 03 20:42:13.078946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:13.078836 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" event={"ID":"3647f3e8-3643-4845-b8ac-b7577b04ba1a","Type":"ContainerStarted","Data":"6990ca051ca0da46fcacaede7998d89357fc0be7e1f749afa432063bcd11bfa1"} Nov 03 20:42:13.175188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:13.175153 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v"] Nov 03 20:42:13.182069 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:13.182038 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bee0c30_a7ba_4da5_8c82_741760c6a2f9.slice/crio-39211b406ca29b8ae644050b4cc1d0108d84175ca34e8bfd2c6b891cb3aabafa WatchSource:0}: Error finding container 39211b406ca29b8ae644050b4cc1d0108d84175ca34e8bfd2c6b891cb3aabafa: Status 404 returned error can't find the container with id 39211b406ca29b8ae644050b4cc1d0108d84175ca34e8bfd2c6b891cb3aabafa Nov 03 20:42:13.202401 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:13.202343 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" podStartSLOduration=1.202326859 podStartE2EDuration="1.202326859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:13.199967407 +0000 UTC m=+741.265208880" watchObservedRunningTime="2026-11-03 20:42:13.202326859 +0000 UTC m=+741.267568336" Nov 03 20:42:14.086097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.086057 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" event={"ID":"3bee0c30-a7ba-4da5-8c82-741760c6a2f9","Type":"ContainerStarted","Data":"bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc"} Nov 03 20:42:14.086486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.086110 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:14.086486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.086127 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" event={"ID":"3bee0c30-a7ba-4da5-8c82-741760c6a2f9","Type":"ContainerStarted","Data":"39211b406ca29b8ae644050b4cc1d0108d84175ca34e8bfd2c6b891cb3aabafa"} Nov 03 20:42:14.086486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.086153 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:14.172918 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:42:14.172865 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:42:14.195498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.195349 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" podStartSLOduration=2.195334023 podStartE2EDuration="2.195334023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:14.194818894 +0000 UTC m=+742.260060367" watchObservedRunningTime="2026-11-03 20:42:14.195334023 +0000 UTC m=+742.260575496" Nov 03 20:42:14.956566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.956523 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb"] Nov 03 20:42:14.961328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:14.961300 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.011606 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.011555 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-504eb0\"" Nov 03 20:42:15.014374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.014351 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb"] Nov 03 20:42:15.020479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.020451 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28dh\" (UniqueName: \"kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh\") pod \"payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.020597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.020498 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.056767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.056740 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-504eb0-dockercfg-cvp8d\"" Nov 03 20:42:15.145676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.144549 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28dh\" (UniqueName: \"kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh\") pod \"payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.145676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.144642 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.146718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.146695 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.202904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.202879 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28dh\" (UniqueName: \"kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh\") pod \"payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.275426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.275353 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:15.670288 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:15.670252 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb"] Nov 03 20:42:15.675567 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:15.675538 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108684b1_6fb1_4d9f_863d_7745c1fe6837.slice/crio-a3a729643e982099046a1249a359ede4a967bde60d6893662198c7f4fc9cc37f WatchSource:0}: Error finding container a3a729643e982099046a1249a359ede4a967bde60d6893662198c7f4fc9cc37f: Status 404 returned error can't find the container with id a3a729643e982099046a1249a359ede4a967bde60d6893662198c7f4fc9cc37f Nov 03 20:42:16.099560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:16.099474 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" event={"ID":"108684b1-6fb1-4d9f-863d-7745c1fe6837","Type":"ContainerStarted","Data":"8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf"} Nov 03 20:42:16.099560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:16.099544 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:16.099735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:16.099573 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" event={"ID":"108684b1-6fb1-4d9f-863d-7745c1fe6837","Type":"ContainerStarted","Data":"a3a729643e982099046a1249a359ede4a967bde60d6893662198c7f4fc9cc37f"} Nov 03 20:42:16.209104 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:16.209049 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" podStartSLOduration=2.20903318 podStartE2EDuration="2.20903318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:16.207425564 +0000 UTC m=+744.272667039" watchObservedRunningTime="2026-11-03 20:42:16.20903318 +0000 UTC m=+744.274274653" Nov 03 20:42:17.037907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.037881 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 20:42:17.454236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.454201 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp"] Nov 03 20:42:17.458423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.458405 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.511181 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.511148 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp"] Nov 03 20:42:17.580670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.579961 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-0cd49b-dockercfg-5fc66\"" Nov 03 20:42:17.580670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.580039 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-0cd49b\"" Nov 03 20:42:17.580923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.580792 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.580923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.580839 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7jk\" (UniqueName: \"kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk\") pod \"payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.598278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.598238 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k"] Nov 03 20:42:17.602265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.602245 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.657104 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.657069 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k"] Nov 03 20:42:17.682183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.682151 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7jk\" (UniqueName: \"kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk\") pod \"payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.682380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.682234 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gtl\" (UniqueName: \"kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl\") pod \"payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.682380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.682331 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.682380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.682367 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.683205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.683177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.741238 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.741151 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7jk\" (UniqueName: \"kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk\") pod \"payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.773867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.773839 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:17.783216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.783189 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gtl\" (UniqueName: \"kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl\") pod \"payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.783345 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.783270 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.784227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.784203 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.842850 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.842323 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gtl\" (UniqueName: \"kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl\") pod \"payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:17.916312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:17.916280 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:18.051571 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:18.051493 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 20:42:18.146467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:18.146438 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp"] Nov 03 20:42:18.152880 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:18.152856 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d6912c_89d8_414e_ba8c_d3bec81b13b1.slice/crio-687b76d47570cc27dbd0d46e676b92ca1911decc4aaf934bd193957a6ba89295 WatchSource:0}: Error finding container 687b76d47570cc27dbd0d46e676b92ca1911decc4aaf934bd193957a6ba89295: Status 404 returned error can't find the container with id 687b76d47570cc27dbd0d46e676b92ca1911decc4aaf934bd193957a6ba89295 Nov 03 20:42:18.301374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:18.301346 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k"] Nov 03 20:42:18.307470 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:18.307445 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d201df_b678_414a_92c6_38192347216d.slice/crio-ca245840e4200f49f3c8aeee9c98d312526dbec4392e31a9197b09acbb3a9dce WatchSource:0}: Error finding container ca245840e4200f49f3c8aeee9c98d312526dbec4392e31a9197b09acbb3a9dce: Status 404 returned error can't find the container with id ca245840e4200f49f3c8aeee9c98d312526dbec4392e31a9197b09acbb3a9dce Nov 03 20:42:19.119407 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.119373 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" event={"ID":"d2d6912c-89d8-414e-ba8c-d3bec81b13b1","Type":"ContainerStarted","Data":"21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c"} Nov 03 20:42:19.119871 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.119429 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" event={"ID":"d2d6912c-89d8-414e-ba8c-d3bec81b13b1","Type":"ContainerStarted","Data":"687b76d47570cc27dbd0d46e676b92ca1911decc4aaf934bd193957a6ba89295"} Nov 03 20:42:19.119871 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.119456 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:19.121019 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.120982 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" event={"ID":"e8d201df-b678-414a-92c6-38192347216d","Type":"ContainerStarted","Data":"c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd"} Nov 03 20:42:19.121138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.121041 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" event={"ID":"e8d201df-b678-414a-92c6-38192347216d","Type":"ContainerStarted","Data":"ca245840e4200f49f3c8aeee9c98d312526dbec4392e31a9197b09acbb3a9dce"} Nov 03 20:42:19.121138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.121070 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:19.228147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.228093 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" podStartSLOduration=2.228076774 podStartE2EDuration="2.228076774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:19.227158561 +0000 UTC m=+747.292400035" watchObservedRunningTime="2026-11-03 20:42:19.228076774 +0000 UTC m=+747.293318248" Nov 03 20:42:19.348847 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:19.348794 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" podStartSLOduration=2.348781512 podStartE2EDuration="2.348781512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:19.348158799 +0000 UTC m=+747.413400274" watchObservedRunningTime="2026-11-03 20:42:19.348781512 +0000 UTC m=+747.414023008" Nov 03 20:42:20.128000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:20.127965 5932 generic.go:356] "Generic (PLEG): container finished" podID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerID="1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892" exitCode=0 Nov 03 20:42:20.128503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:20.128034 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" event={"ID":"c4826cc8-9dd1-4e79-a2ba-8df64125e818","Type":"ContainerDied","Data":"1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892"} Nov 03 20:42:20.128503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:20.128395 5932 scope.go:122] "RemoveContainer" containerID="1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892" Nov 03 20:42:20.129654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:20.129614 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:20.129757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:20.129690 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:20.129757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:20.129722 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:21.152387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:21.152355 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" event={"ID":"c4826cc8-9dd1-4e79-a2ba-8df64125e818","Type":"ContainerStarted","Data":"36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b"} Nov 03 20:42:22.074356 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.074329 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 20:42:22.074534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.074508 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 20:42:22.146109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.146071 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:42:22.146935 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.146909 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:42:22.147088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.146966 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 20:42:22.534646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.520993 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2"] Nov 03 20:42:22.546045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.546016 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.578531 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.578493 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2"] Nov 03 20:42:22.594833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.594810 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-2a5ec9-dockercfg-kkczl\"" Nov 03 20:42:22.594833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.594817 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-2a5ec9\"" Nov 03 20:42:22.656088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.656057 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.656088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.656092 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xfqj\" (UniqueName: \"kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj\") pod \"payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.757711 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.757664 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.758008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.757721 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xfqj\" (UniqueName: \"kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj\") pod \"payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.759101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.759068 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.815661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.815578 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xfqj\" (UniqueName: \"kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj\") pod \"payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:22.859279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:22.859247 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:23.145698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:23.138006 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:42:23.145698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:23.138067 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 20:42:23.247227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:23.247190 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2"] Nov 03 20:42:23.250310 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:42:23.250280 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310b02f1_14ef_4dd5_a2e6_fc2b54c0e0f0.slice/crio-3d2218df283ecfdcee8be713b620008e75eb5609ddda4bb45f75c8329f79f2bc WatchSource:0}: Error finding container 3d2218df283ecfdcee8be713b620008e75eb5609ddda4bb45f75c8329f79f2bc: Status 404 returned error can't find the container with id 3d2218df283ecfdcee8be713b620008e75eb5609ddda4bb45f75c8329f79f2bc Nov 03 20:42:23.954575 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:23.954523 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:42:24.145788 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:24.145747 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:42:24.145983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:24.145815 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 20:42:24.175694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:24.175662 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" event={"ID":"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0","Type":"ContainerStarted","Data":"7cad49f3fc874c19eac9b87dff1cdadc7d56f7fdb416989e04714f8845cdebd1"} Nov 03 20:42:24.175869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:24.175722 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" event={"ID":"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0","Type":"ContainerStarted","Data":"3d2218df283ecfdcee8be713b620008e75eb5609ddda4bb45f75c8329f79f2bc"} Nov 03 20:42:24.175869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:24.175751 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:24.302711 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:24.302598 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" podStartSLOduration=2.302584178 podStartE2EDuration="2.302584178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:42:24.301315846 +0000 UTC m=+752.366557331" watchObservedRunningTime="2026-11-03 20:42:24.302584178 +0000 UTC m=+752.367825651" Nov 03 20:42:25.094516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:25.094486 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 20:42:25.094969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:25.094678 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 20:42:25.142768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:25.142724 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:42:25.143043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:25.143015 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 20:42:26.136872 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:26.136843 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 20:42:26.137276 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:26.136903 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 20:42:27.109963 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:27.109934 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 20:42:27.140848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:27.140816 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:42:27.141775 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:27.141756 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 20:42:30.130988 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:30.130959 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 20:42:30.131363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:30.131028 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 20:42:35.185008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:35.184979 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 20:42:44.150846 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:42:44.150808 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:42:44.307659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:44.307546 5932 generic.go:356] "Generic (PLEG): container finished" podID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerID="0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e" exitCode=0 Nov 03 20:42:44.307659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:44.307607 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" event={"ID":"051ea739-77c3-43ae-bcda-7d1915deeb20","Type":"ContainerDied","Data":"0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e"} Nov 03 20:42:44.308045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:44.308027 5932 scope.go:122] "RemoveContainer" containerID="0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e" Nov 03 20:42:44.309525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:44.309493 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:44.309669 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:44.309568 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:44.309669 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:44.309600 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:45.315915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:45.315882 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" event={"ID":"051ea739-77c3-43ae-bcda-7d1915deeb20","Type":"ContainerStarted","Data":"d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4"} Nov 03 20:42:45.410036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:45.409994 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:42:45.410231 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:45.410147 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:42:45.410231 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:45.410204 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:42:46.410455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:46.410423 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:42:46.410887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:46.410470 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:42:47.411089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:47.411054 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:42:47.411463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:47.411114 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:42:48.410901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:48.410871 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:42:48.411141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:48.411114 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:42:49.410717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:49.410688 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:42:49.410945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:49.410924 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:42:50.410566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:50.410530 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 20:42:50.410971 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:50.410594 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 20:42:51.414740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:51.414713 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:42:51.415211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:51.415192 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:42:51.415807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:51.415790 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 20:42:55.490160 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:55.490121 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:44908->10.132.0.61:15021: read: connection reset by peer" start-of-body= Nov 03 20:42:55.490649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:55.490184 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": read tcp 10.132.0.2:44908->10.132.0.61:15021: read: connection reset by peer" Nov 03 20:42:56.435884 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:56.435850 5932 generic.go:356] "Generic (PLEG): container finished" podID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerID="60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03" exitCode=0 Nov 03 20:42:56.436083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:56.435928 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" event={"ID":"6f9a630a-9407-40e4-b964-61b39dd48e75","Type":"ContainerDied","Data":"60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03"} Nov 03 20:42:56.436338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:56.436320 5932 scope.go:122] "RemoveContainer" containerID="60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03" Nov 03 20:42:56.437617 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:56.437594 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:56.437694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:56.437674 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:56.437734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:56.437705 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:42:57.456498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:57.456464 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" event={"ID":"6f9a630a-9407-40e4-b964-61b39dd48e75","Type":"ContainerStarted","Data":"d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa"} Nov 03 20:42:58.450493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:58.450461 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:42:58.450725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:58.450643 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:42:58.450725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:58.450686 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:42:59.450783 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:59.450737 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:42:59.451206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:42:59.450799 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:43:00.450499 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:00.450460 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:43:00.450727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:00.450539 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:43:01.450942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:01.450905 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:43:01.451356 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:01.450965 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:43:02.451325 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:02.451292 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 20:43:02.451804 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:02.451357 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 20:43:03.454805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:03.454776 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:43:03.455304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:03.455054 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:43:03.455823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:03.455804 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 20:43:14.152921 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:43:14.152882 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:43:44.150131 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:43:44.150056 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:43:52.386213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.386185 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb"] Nov 03 20:43:52.390099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.390081 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.418213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.418181 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.418344 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.418266 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.418392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.418364 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.418443 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.418392 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.418515 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.418493 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.418582 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.418546 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjclb\" (UniqueName: \"kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.447956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.447928 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb"] Nov 03 20:43:52.488409 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.488380 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"kube-root-ca.crt\"" Nov 03 20:43:52.488615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.488596 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"simulator-w5-2a5ec9-kserve-self-signed-certs\"" Nov 03 20:43:52.488699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.488596 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"openshift-service-ca.crt\"" Nov 03 20:43:52.488699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.488647 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"default-dockercfg-9dx65\"" Nov 03 20:43:52.535277 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.535241 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.535489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.535335 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.535489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.535361 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.535489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.535424 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.535489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.535455 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjclb\" (UniqueName: \"kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.536475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.535560 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.536475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.536135 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.536475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.536460 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.539789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.539761 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.542462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.542443 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.543097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.543078 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.601491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.601461 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjclb\" (UniqueName: \"kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb\") pod \"simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:52.709907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:52.709845 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:43:53.086562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:53.086468 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb"] Nov 03 20:43:53.091417 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:43:53.091393 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e830b87_2d2a_4d61_9171_82ce54bdba23.slice/crio-c8ad4be10566ba0d57d573be43d007ead2781539387ecb6d7a9cd7726b64e54b WatchSource:0}: Error finding container c8ad4be10566ba0d57d573be43d007ead2781539387ecb6d7a9cd7726b64e54b: Status 404 returned error can't find the container with id c8ad4be10566ba0d57d573be43d007ead2781539387ecb6d7a9cd7726b64e54b Nov 03 20:43:53.820030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:53.819997 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" event={"ID":"2e830b87-2d2a-4d61-9171-82ce54bdba23","Type":"ContainerStarted","Data":"ba3e4672fce56e0ce5e08b18377d0f522e2976d1bce3c7ad53dc224078cce6f5"} Nov 03 20:43:53.820030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:53.820036 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" event={"ID":"2e830b87-2d2a-4d61-9171-82ce54bdba23","Type":"ContainerStarted","Data":"c8ad4be10566ba0d57d573be43d007ead2781539387ecb6d7a9cd7726b64e54b"} Nov 03 20:43:53.929826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:53.929774 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podStartSLOduration=1.92976145 podStartE2EDuration="1.92976145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:43:53.929001706 +0000 UTC m=+841.994243180" watchObservedRunningTime="2026-11-03 20:43:53.92976145 +0000 UTC m=+841.995002923" Nov 03 20:43:54.710420 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:43:54.710386 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:44:02.710494 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:02.710461 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:44:02.723224 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:02.723204 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:44:02.882670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:02.882646 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:44:14.161507 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:44:14.161458 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:44:17.065010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.064977 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf"] Nov 03 20:44:17.069216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.069196 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.083810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.083785 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.083929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.083818 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.083929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.083846 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmcmj\" (UniqueName: \"kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.084001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.083979 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.084032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.084008 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.084073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.084036 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.118867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.118827 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"openshift-service-ca.crt\"" Nov 03 20:44:17.119045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.118876 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"kube-root-ca.crt\"" Nov 03 20:44:17.165337 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.165307 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"default-dockercfg-kzx24\"" Nov 03 20:44:17.167893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.167867 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf"] Nov 03 20:44:17.168446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.168429 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"simulator-w1-0cd49b-kserve-self-signed-certs\"" Nov 03 20:44:17.184749 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.184723 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.184870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.184764 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.184870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.184807 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmcmj\" (UniqueName: \"kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.184984 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.184872 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.184984 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.184899 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.185089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.185032 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.185195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.185172 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.185263 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.185204 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.185313 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.185259 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.187056 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.187030 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.187358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.187339 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.243412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.243384 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmcmj\" (UniqueName: \"kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj\") pod \"simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.382516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.382484 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:17.762854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.762761 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf"] Nov 03 20:44:17.769216 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:44:17.769183 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd41fe606_bbc7_4223_aa45_ab49d2b5fbc8.slice/crio-e5495b132549ee118e1c3a4ed37eed778537aec453bc08ac3223689704a01c39 WatchSource:0}: Error finding container e5495b132549ee118e1c3a4ed37eed778537aec453bc08ac3223689704a01c39: Status 404 returned error can't find the container with id e5495b132549ee118e1c3a4ed37eed778537aec453bc08ac3223689704a01c39 Nov 03 20:44:17.970278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.970246 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" event={"ID":"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8","Type":"ContainerStarted","Data":"a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663"} Nov 03 20:44:17.970459 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:17.970287 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" event={"ID":"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8","Type":"ContainerStarted","Data":"e5495b132549ee118e1c3a4ed37eed778537aec453bc08ac3223689704a01c39"} Nov 03 20:44:18.080656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:18.080527 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podStartSLOduration=1.080512942 podStartE2EDuration="1.080512942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:18.079065104 +0000 UTC m=+866.144306576" watchObservedRunningTime="2026-11-03 20:44:18.080512942 +0000 UTC m=+866.145754414" Nov 03 20:44:18.383096 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:18.383066 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:27.382922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:27.382881 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:27.395884 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:27.395860 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:28.056549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:28.056518 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:44:44.155357 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:44:44.155317 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:44:52.220161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.220129 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:44:52.223660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.223634 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:44:52.903099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.903063 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c"] Nov 03 20:44:52.907666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.907610 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975180 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxxb\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975218 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975260 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975351 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975387 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975403 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975545 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975573 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:52.975602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:52.975589 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.002576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.002539 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-88228a-openshift-default-dockercfg-tb65c\"" Nov 03 20:44:53.005739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.005712 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c"] Nov 03 20:44:53.077204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077158 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077203 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077220 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077250 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxxb\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077278 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077342 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077384 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077421 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.077660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077444 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.078083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.077923 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.078196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.078160 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.078339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.078197 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.078475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.078445 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.078596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.078519 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.080355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.080328 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.080875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.080846 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.134910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.134876 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.135126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.135104 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxxb\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.223599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.223508 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:53.657758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.657725 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c"] Nov 03 20:44:53.664029 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:44:53.664001 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9ee5ad_f460_40e9_9a88_721b89da9f86.slice/crio-ea6bccf32cf3cce76e1d8113421e1a94e502cf00b2d9c9776a7b781ff8ee1bb6 WatchSource:0}: Error finding container ea6bccf32cf3cce76e1d8113421e1a94e502cf00b2d9c9776a7b781ff8ee1bb6: Status 404 returned error can't find the container with id ea6bccf32cf3cce76e1d8113421e1a94e502cf00b2d9c9776a7b781ff8ee1bb6 Nov 03 20:44:53.666472 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.666436 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:44:53.666581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.666509 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:44:53.666581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:53.666545 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:44:54.207556 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:54.207514 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" event={"ID":"dd9ee5ad-f460-40e9-9a88-721b89da9f86","Type":"ContainerStarted","Data":"6f0d9797c5a41f15e43f102ee40da2a23aaf674899d3f8323d486983d6381812"} Nov 03 20:44:54.207772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:54.207578 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" event={"ID":"dd9ee5ad-f460-40e9-9a88-721b89da9f86","Type":"ContainerStarted","Data":"ea6bccf32cf3cce76e1d8113421e1a94e502cf00b2d9c9776a7b781ff8ee1bb6"} Nov 03 20:44:54.233648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:54.229090 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:44:54.233648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:54.229162 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:44:54.233648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:54.229458 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:44:54.351647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:54.349126 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podStartSLOduration=2.349102822 podStartE2EDuration="2.349102822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:44:54.320958335 +0000 UTC m=+902.386199809" watchObservedRunningTime="2026-11-03 20:44:54.349102822 +0000 UTC m=+902.414344298" Nov 03 20:44:55.225421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:55.224991 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:44:55.225421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:55.225050 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:44:56.224503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:56.224051 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:44:56.224503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:56.224115 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:44:57.233064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:57.233026 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:44:57.233692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:57.233653 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:44:58.225648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:58.225153 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:44:58.225648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:58.225234 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:44:59.232121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:59.232080 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:44:59.232766 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:44:59.232728 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 20:45:00.186983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.186938 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7"] Nov 03 20:45:00.192243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.192213 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh"] Nov 03 20:45:00.192390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.192314 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.196081 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.196057 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl"] Nov 03 20:45:00.196234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.196190 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.199819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.199791 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.233371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.233340 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls"] Nov 03 20:45:00.238211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.238184 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5"] Nov 03 20:45:00.238362 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.238327 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.242549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.242522 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k"] Nov 03 20:45:00.242710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.242681 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.244755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.244732 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:45:00.246287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.246269 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.291757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.291726 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.291965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.291786 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4b6\" (UniqueName: \"kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.291965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.291945 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7df\" (UniqueName: \"kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.292107 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.292037 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.292107 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.292074 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.292225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.292185 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwfc\" (UniqueName: \"kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.295953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.295924 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7"] Nov 03 20:45:00.302032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.302000 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh"] Nov 03 20:45:00.308098 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.306203 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls"] Nov 03 20:45:00.352321 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.331157 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl"] Nov 03 20:45:00.352321 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.334536 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5"] Nov 03 20:45:00.355681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.355649 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k"] Nov 03 20:45:00.392790 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.392753 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwfc\" (UniqueName: \"kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.393003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.392802 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9js8t\" (UniqueName: \"kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.393003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.392827 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.393003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.392984 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.393189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393072 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4b6\" (UniqueName: \"kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.393189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393129 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnhc5\" (UniqueName: \"kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.393189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393166 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.393347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393277 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7df\" (UniqueName: \"kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.393347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393324 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.393455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393350 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.393455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393384 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.393455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393437 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5z58\" (UniqueName: \"kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.393887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393861 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.394016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.393997 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.394091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.394060 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.457113 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.457038 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwfc\" (UniqueName: \"kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc\") pod \"maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.457278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.457216 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4b6\" (UniqueName: \"kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.457338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.457317 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7df\" (UniqueName: \"kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df\") pod \"maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.494373 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.494339 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9js8t\" (UniqueName: \"kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.494373 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.494376 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.494596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.494431 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnhc5\" (UniqueName: \"kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.494596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.494451 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.494737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.494587 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.494737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.494689 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5z58\" (UniqueName: \"kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.495218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.495177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.495218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.495206 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.495471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.495450 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.509570 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.509531 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:00.517709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.517677 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:00.524756 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.524723 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:00.553053 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.553020 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnhc5\" (UniqueName: \"kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.558080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.558046 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5z58\" (UniqueName: \"kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.559131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.559104 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9js8t\" (UniqueName: \"kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.567176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.566607 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:00.580324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.580181 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:00.854888 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.854853 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:00.906637 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.906551 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7"] Nov 03 20:45:00.913955 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:00.913916 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b1d05e_b502_47f0_9f50_370d8be91728.slice/crio-f4fac8675a05fef77292f307c895031b8f884d8a90a3fcb70f7548410f44ba57 WatchSource:0}: Error finding container f4fac8675a05fef77292f307c895031b8f884d8a90a3fcb70f7548410f44ba57: Status 404 returned error can't find the container with id f4fac8675a05fef77292f307c895031b8f884d8a90a3fcb70f7548410f44ba57 Nov 03 20:45:00.954061 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.954016 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh"] Nov 03 20:45:00.964114 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:00.964000 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd940fd6e_a7fa_43a1_ac47_ca1ebfb1d1de.slice/crio-668a4e0ae30678024f8ff7075c3eafe2c8ebab259700025dc86e817fb92da690 WatchSource:0}: Error finding container 668a4e0ae30678024f8ff7075c3eafe2c8ebab259700025dc86e817fb92da690: Status 404 returned error can't find the container with id 668a4e0ae30678024f8ff7075c3eafe2c8ebab259700025dc86e817fb92da690 Nov 03 20:45:00.973396 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:00.968049 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl"] Nov 03 20:45:00.977839 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:00.976887 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2d030d_70d7_4162_bab8_0f0bec346b8a.slice/crio-7e488f792847ec4077f7ef3ff10fd965e1bd2cba5137f53c47dc853c8424cc2c WatchSource:0}: Error finding container 7e488f792847ec4077f7ef3ff10fd965e1bd2cba5137f53c47dc853c8424cc2c: Status 404 returned error can't find the container with id 7e488f792847ec4077f7ef3ff10fd965e1bd2cba5137f53c47dc853c8424cc2c Nov 03 20:45:01.029028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.028985 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5"] Nov 03 20:45:01.033072 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.033032 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k"] Nov 03 20:45:01.037617 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:01.037583 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1c8b6a_6a98_44b6_aae9_13a517e734f5.slice/crio-0173d210a873f0f0b3ccac47b07cee8bfae940ae08000166d600f8340bf299d8 WatchSource:0}: Error finding container 0173d210a873f0f0b3ccac47b07cee8bfae940ae08000166d600f8340bf299d8: Status 404 returned error can't find the container with id 0173d210a873f0f0b3ccac47b07cee8bfae940ae08000166d600f8340bf299d8 Nov 03 20:45:01.039557 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:01.039533 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3b0293_44d4_4b5b_8ad8_1c50ae391b1e.slice/crio-b3963177b181b97592562f25874e3b88195873df5ab7cdec62f09555375e9f09 WatchSource:0}: Error finding container b3963177b181b97592562f25874e3b88195873df5ab7cdec62f09555375e9f09: Status 404 returned error can't find the container with id b3963177b181b97592562f25874e3b88195873df5ab7cdec62f09555375e9f09 Nov 03 20:45:01.248895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.248803 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls"] Nov 03 20:45:01.254852 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:01.254807 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1639bf18_6962_45a1_90b9_00c5431ded13.slice/crio-9e3354668a3c7b00f1bd09e14c04e612c0de90bfd61bc6cc3dab08fb300649a5 WatchSource:0}: Error finding container 9e3354668a3c7b00f1bd09e14c04e612c0de90bfd61bc6cc3dab08fb300649a5: Status 404 returned error can't find the container with id 9e3354668a3c7b00f1bd09e14c04e612c0de90bfd61bc6cc3dab08fb300649a5 Nov 03 20:45:01.266782 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.266751 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:45:01.266903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.266812 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:45:01.282942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.282905 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerStarted","Data":"9e3354668a3c7b00f1bd09e14c04e612c0de90bfd61bc6cc3dab08fb300649a5"} Nov 03 20:45:01.284363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.284333 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerStarted","Data":"7e488f792847ec4077f7ef3ff10fd965e1bd2cba5137f53c47dc853c8424cc2c"} Nov 03 20:45:01.285647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.285595 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerStarted","Data":"f4fac8675a05fef77292f307c895031b8f884d8a90a3fcb70f7548410f44ba57"} Nov 03 20:45:01.286902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.286873 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerStarted","Data":"b3963177b181b97592562f25874e3b88195873df5ab7cdec62f09555375e9f09"} Nov 03 20:45:01.288279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.288257 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" event={"ID":"cf1c8b6a-6a98-44b6-aae9-13a517e734f5","Type":"ContainerStarted","Data":"0173d210a873f0f0b3ccac47b07cee8bfae940ae08000166d600f8340bf299d8"} Nov 03 20:45:01.289528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.289509 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerStarted","Data":"668a4e0ae30678024f8ff7075c3eafe2c8ebab259700025dc86e817fb92da690"} Nov 03 20:45:01.340562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.340531 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:45:01.341036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.341016 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:45:01.341889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:01.341868 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:45:07.348579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.348543 5932 generic.go:356] "Generic (PLEG): container finished" podID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerID="d8be3c8ea5e3cfa9cd319ff79ba9a0f6e169afd9cd5775ae2eb56d464f72c368" exitCode=60 Nov 03 20:45:07.348968 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.348618 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerDied","Data":"d8be3c8ea5e3cfa9cd319ff79ba9a0f6e169afd9cd5775ae2eb56d464f72c368"} Nov 03 20:45:07.348968 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.348888 5932 scope.go:122] "RemoveContainer" containerID="d8be3c8ea5e3cfa9cd319ff79ba9a0f6e169afd9cd5775ae2eb56d464f72c368" Nov 03 20:45:07.350353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.350322 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerStarted","Data":"0d32aa828c9be263c502cac88a70076158b2004b6287eb1bc9f58ebafa0f1544"} Nov 03 20:45:07.352035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.352015 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerStarted","Data":"6b22942364918a5b1ad79d573f15876c799cd98e541062e887b1bb884149ee10"} Nov 03 20:45:07.353679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.353654 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerStarted","Data":"08690f35c78469ab452d43612f8e1fb8135f13801326805114784232a9ce9567"} Nov 03 20:45:07.360103 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.359775 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerStarted","Data":"dd4b300c6d91931315090b0128ffbdf596c182d085bb6a73d6387a35eb43c3c2"} Nov 03 20:45:07.361424 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.361404 5932 generic.go:356] "Generic (PLEG): container finished" podID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerID="8547e21411417960904b93db5da8806676bd5217f942b084d424102633cb204a" exitCode=60 Nov 03 20:45:07.361530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.361484 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" event={"ID":"cf1c8b6a-6a98-44b6-aae9-13a517e734f5","Type":"ContainerDied","Data":"8547e21411417960904b93db5da8806676bd5217f942b084d424102633cb204a"} Nov 03 20:45:07.361631 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.361605 5932 scope.go:122] "RemoveContainer" containerID="8547e21411417960904b93db5da8806676bd5217f942b084d424102633cb204a" Nov 03 20:45:07.572472 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.572418 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" podStartSLOduration=1.9866633870000001 podStartE2EDuration="7.572403011s" totalImagesPullingTime="5.585739624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:07.570099619 +0000 UTC m=+915.635341093" watchObservedRunningTime="2026-11-03 20:45:07.572403011 +0000 UTC m=+915.637644481" Nov 03 20:45:07.678934 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.678880 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" podStartSLOduration=2.267124776 podStartE2EDuration="7.678864407s" totalImagesPullingTime="5.411739631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:07.678041532 +0000 UTC m=+915.743283028" watchObservedRunningTime="2026-11-03 20:45:07.678864407 +0000 UTC m=+915.744105877" Nov 03 20:45:07.787204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:07.787126 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" podStartSLOduration=2.412058839 podStartE2EDuration="7.787111597s" totalImagesPullingTime="5.375052758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:07.784949558 +0000 UTC m=+915.850191030" watchObservedRunningTime="2026-11-03 20:45:07.787111597 +0000 UTC m=+915.852353112" Nov 03 20:45:08.004390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.004333 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" podStartSLOduration=2.247879862 podStartE2EDuration="8.004318512s" totalImagesPullingTime="5.75643865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:08.001292917 +0000 UTC m=+916.066534392" watchObservedRunningTime="2026-11-03 20:45:08.004318512 +0000 UTC m=+916.069559985" Nov 03 20:45:08.370347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.370314 5932 generic.go:356] "Generic (PLEG): container finished" podID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerID="7f50d2ca32dfaaf46a718c496bb7340652266663bc75b6d39884a5ff9fb260b2" exitCode=60 Nov 03 20:45:08.370801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.370384 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" event={"ID":"cf1c8b6a-6a98-44b6-aae9-13a517e734f5","Type":"ContainerDied","Data":"7f50d2ca32dfaaf46a718c496bb7340652266663bc75b6d39884a5ff9fb260b2"} Nov 03 20:45:08.370801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.370468 5932 scope.go:122] "RemoveContainer" containerID="8547e21411417960904b93db5da8806676bd5217f942b084d424102633cb204a" Nov 03 20:45:08.370801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.370733 5932 scope.go:122] "RemoveContainer" containerID="7f50d2ca32dfaaf46a718c496bb7340652266663bc75b6d39884a5ff9fb260b2" Nov 03 20:45:08.371042 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:08.371021 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k_odh-ai-gateway-infra(cf1c8b6a-6a98-44b6-aae9-13a517e734f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" Nov 03 20:45:08.372518 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.372494 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerStarted","Data":"d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4"} Nov 03 20:45:08.374059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.374038 5932 generic.go:356] "Generic (PLEG): container finished" podID="1639bf18-6962-45a1-90b9-00c5431ded13" containerID="0d32aa828c9be263c502cac88a70076158b2004b6287eb1bc9f58ebafa0f1544" exitCode=60 Nov 03 20:45:08.374167 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.374107 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerDied","Data":"0d32aa828c9be263c502cac88a70076158b2004b6287eb1bc9f58ebafa0f1544"} Nov 03 20:45:08.374279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.374265 5932 scope.go:122] "RemoveContainer" containerID="0d32aa828c9be263c502cac88a70076158b2004b6287eb1bc9f58ebafa0f1544" Nov 03 20:45:08.375788 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.375767 5932 generic.go:356] "Generic (PLEG): container finished" podID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerID="6b22942364918a5b1ad79d573f15876c799cd98e541062e887b1bb884149ee10" exitCode=60 Nov 03 20:45:08.375898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.375833 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerDied","Data":"6b22942364918a5b1ad79d573f15876c799cd98e541062e887b1bb884149ee10"} Nov 03 20:45:08.376083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.376064 5932 scope.go:122] "RemoveContainer" containerID="6b22942364918a5b1ad79d573f15876c799cd98e541062e887b1bb884149ee10" Nov 03 20:45:08.377693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.377671 5932 generic.go:356] "Generic (PLEG): container finished" podID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerID="08690f35c78469ab452d43612f8e1fb8135f13801326805114784232a9ce9567" exitCode=60 Nov 03 20:45:08.377786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.377748 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerDied","Data":"08690f35c78469ab452d43612f8e1fb8135f13801326805114784232a9ce9567"} Nov 03 20:45:08.377917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.377903 5932 scope.go:122] "RemoveContainer" containerID="08690f35c78469ab452d43612f8e1fb8135f13801326805114784232a9ce9567" Nov 03 20:45:08.380078 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.380049 5932 generic.go:356] "Generic (PLEG): container finished" podID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerID="dd4b300c6d91931315090b0128ffbdf596c182d085bb6a73d6387a35eb43c3c2" exitCode=60 Nov 03 20:45:08.380174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.380097 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerDied","Data":"dd4b300c6d91931315090b0128ffbdf596c182d085bb6a73d6387a35eb43c3c2"} Nov 03 20:45:08.380312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.380302 5932 scope.go:122] "RemoveContainer" containerID="dd4b300c6d91931315090b0128ffbdf596c182d085bb6a73d6387a35eb43c3c2" Nov 03 20:45:08.712182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:08.712133 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" podStartSLOduration=3.267239374 podStartE2EDuration="8.712119604s" totalImagesPullingTime="5.44488023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:08.709356362 +0000 UTC m=+916.774597834" watchObservedRunningTime="2026-11-03 20:45:08.712119604 +0000 UTC m=+916.777361077" Nov 03 20:45:09.389033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.389002 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerStarted","Data":"efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b"} Nov 03 20:45:09.390981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.390960 5932 scope.go:122] "RemoveContainer" containerID="7f50d2ca32dfaaf46a718c496bb7340652266663bc75b6d39884a5ff9fb260b2" Nov 03 20:45:09.391232 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:09.391208 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k_odh-ai-gateway-infra(cf1c8b6a-6a98-44b6-aae9-13a517e734f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" Nov 03 20:45:09.392679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.392657 5932 generic.go:356] "Generic (PLEG): container finished" podID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerID="d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4" exitCode=60 Nov 03 20:45:09.392768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.392750 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerDied","Data":"d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4"} Nov 03 20:45:09.392812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.392794 5932 scope.go:122] "RemoveContainer" containerID="d8be3c8ea5e3cfa9cd319ff79ba9a0f6e169afd9cd5775ae2eb56d464f72c368" Nov 03 20:45:09.392987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.392970 5932 scope.go:122] "RemoveContainer" containerID="d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4" Nov 03 20:45:09.393198 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:09.393176 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh_odh-ai-gateway-infra(d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" Nov 03 20:45:09.394836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.394811 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerStarted","Data":"2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa"} Nov 03 20:45:09.396921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.396902 5932 generic.go:356] "Generic (PLEG): container finished" podID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerID="8e724becbd5ebc55be6a7a716246598f047711d9edcf2a0fd6da81622066871e" exitCode=60 Nov 03 20:45:09.397009 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.396988 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerDied","Data":"8e724becbd5ebc55be6a7a716246598f047711d9edcf2a0fd6da81622066871e"} Nov 03 20:45:09.397217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.397198 5932 scope.go:122] "RemoveContainer" containerID="8e724becbd5ebc55be6a7a716246598f047711d9edcf2a0fd6da81622066871e" Nov 03 20:45:09.397421 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:09.397399 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl_odh-ai-gateway-infra(1b2d030d-70d7-4162-bab8-0f0bec346b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" Nov 03 20:45:09.398876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.398848 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerStarted","Data":"8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b"} Nov 03 20:45:09.405838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:09.405808 5932 scope.go:122] "RemoveContainer" containerID="6b22942364918a5b1ad79d573f15876c799cd98e541062e887b1bb884149ee10" Nov 03 20:45:10.409377 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.409347 5932 scope.go:122] "RemoveContainer" containerID="d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4" Nov 03 20:45:10.409839 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:10.409686 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh_odh-ai-gateway-infra(d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" Nov 03 20:45:10.411223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.411196 5932 generic.go:356] "Generic (PLEG): container finished" podID="1639bf18-6962-45a1-90b9-00c5431ded13" containerID="2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa" exitCode=60 Nov 03 20:45:10.411336 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.411285 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerDied","Data":"2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa"} Nov 03 20:45:10.411391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.411349 5932 scope.go:122] "RemoveContainer" containerID="0d32aa828c9be263c502cac88a70076158b2004b6287eb1bc9f58ebafa0f1544" Nov 03 20:45:10.411508 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.411490 5932 scope.go:122] "RemoveContainer" containerID="2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa" Nov 03 20:45:10.412159 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:10.412132 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls_odh-ai-gateway-infra(1639bf18-6962-45a1-90b9-00c5431ded13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" Nov 03 20:45:10.413336 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.413319 5932 scope.go:122] "RemoveContainer" containerID="8e724becbd5ebc55be6a7a716246598f047711d9edcf2a0fd6da81622066871e" Nov 03 20:45:10.413530 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:10.413513 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl_odh-ai-gateway-infra(1b2d030d-70d7-4162-bab8-0f0bec346b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" Nov 03 20:45:10.415144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.415123 5932 generic.go:356] "Generic (PLEG): container finished" podID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerID="8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b" exitCode=60 Nov 03 20:45:10.415249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.415202 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerDied","Data":"8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b"} Nov 03 20:45:10.415421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.415400 5932 scope.go:122] "RemoveContainer" containerID="8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b" Nov 03 20:45:10.415656 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:10.415633 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7_odh-ai-gateway-infra(a2b1d05e-b502-47f0-9f50-370d8be91728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" Nov 03 20:45:10.416980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.416960 5932 generic.go:356] "Generic (PLEG): container finished" podID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerID="efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b" exitCode=60 Nov 03 20:45:10.417061 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.417028 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerDied","Data":"efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b"} Nov 03 20:45:10.417247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.417235 5932 scope.go:122] "RemoveContainer" containerID="efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b" Nov 03 20:45:10.417432 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:10.417418 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5_odh-ai-gateway-infra(6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" Nov 03 20:45:10.424059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.424043 5932 scope.go:122] "RemoveContainer" containerID="08690f35c78469ab452d43612f8e1fb8135f13801326805114784232a9ce9567" Nov 03 20:45:10.436463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:10.436450 5932 scope.go:122] "RemoveContainer" containerID="dd4b300c6d91931315090b0128ffbdf596c182d085bb6a73d6387a35eb43c3c2" Nov 03 20:45:11.424930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:11.424898 5932 scope.go:122] "RemoveContainer" containerID="2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa" Nov 03 20:45:11.425346 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:11.425183 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls_odh-ai-gateway-infra(1639bf18-6962-45a1-90b9-00c5431ded13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" Nov 03 20:45:11.426477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:11.426446 5932 scope.go:122] "RemoveContainer" containerID="8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b" Nov 03 20:45:11.426750 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:11.426730 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7_odh-ai-gateway-infra(a2b1d05e-b502-47f0-9f50-370d8be91728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" Nov 03 20:45:11.427867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:11.427851 5932 scope.go:122] "RemoveContainer" containerID="efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b" Nov 03 20:45:11.428069 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:11.428052 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5_odh-ai-gateway-infra(6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" Nov 03 20:45:14.150778 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:14.150741 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:45:20.045019 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:20.044983 5932 scope.go:122] "RemoveContainer" containerID="efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b" Nov 03 20:45:20.045534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:20.045166 5932 scope.go:122] "RemoveContainer" containerID="8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b" Nov 03 20:45:20.045534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:20.045320 5932 scope.go:122] "RemoveContainer" containerID="d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4" Nov 03 20:45:20.489966 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:20.489923 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerStarted","Data":"65ea0216a1a9eb6dee79b0533a7803b2e540ba21e7c1d3f62b0f02541551a8c4"} Nov 03 20:45:20.491980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:20.491955 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerStarted","Data":"fa576f4f7ba6abf9593246f838314b3ad7fe3e92ca1346fbc94156f0b0db70db"} Nov 03 20:45:20.493758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:20.493738 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerStarted","Data":"7d35493e34716dedcf6bb93978f00b92b63f3e1421c3510d6f1e322a8cff1cbb"} Nov 03 20:45:21.044210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.044185 5932 scope.go:122] "RemoveContainer" containerID="8e724becbd5ebc55be6a7a716246598f047711d9edcf2a0fd6da81622066871e" Nov 03 20:45:21.501670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.501582 5932 generic.go:356] "Generic (PLEG): container finished" podID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerID="65ea0216a1a9eb6dee79b0533a7803b2e540ba21e7c1d3f62b0f02541551a8c4" exitCode=60 Nov 03 20:45:21.502079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.501658 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerDied","Data":"65ea0216a1a9eb6dee79b0533a7803b2e540ba21e7c1d3f62b0f02541551a8c4"} Nov 03 20:45:21.502079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.501723 5932 scope.go:122] "RemoveContainer" containerID="d487ee9fcea29ff53b8f79b76b4a5a684ede4b4040378570460414f23cf2ace4" Nov 03 20:45:21.502079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.502026 5932 scope.go:122] "RemoveContainer" containerID="65ea0216a1a9eb6dee79b0533a7803b2e540ba21e7c1d3f62b0f02541551a8c4" Nov 03 20:45:21.502295 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:21.502275 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh_odh-ai-gateway-infra(d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" Nov 03 20:45:21.503846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.503824 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerStarted","Data":"421c575f961602b20e9611fe30e41db7b1aa75a1246b24faac2cd619dbd79c0d"} Nov 03 20:45:21.505543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.505525 5932 generic.go:356] "Generic (PLEG): container finished" podID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerID="fa576f4f7ba6abf9593246f838314b3ad7fe3e92ca1346fbc94156f0b0db70db" exitCode=60 Nov 03 20:45:21.505644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.505593 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerDied","Data":"fa576f4f7ba6abf9593246f838314b3ad7fe3e92ca1346fbc94156f0b0db70db"} Nov 03 20:45:21.505896 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.505876 5932 scope.go:122] "RemoveContainer" containerID="fa576f4f7ba6abf9593246f838314b3ad7fe3e92ca1346fbc94156f0b0db70db" Nov 03 20:45:21.506136 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:21.506115 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7_odh-ai-gateway-infra(a2b1d05e-b502-47f0-9f50-370d8be91728)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" Nov 03 20:45:21.507476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.507454 5932 generic.go:356] "Generic (PLEG): container finished" podID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerID="7d35493e34716dedcf6bb93978f00b92b63f3e1421c3510d6f1e322a8cff1cbb" exitCode=60 Nov 03 20:45:21.507545 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.507518 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerDied","Data":"7d35493e34716dedcf6bb93978f00b92b63f3e1421c3510d6f1e322a8cff1cbb"} Nov 03 20:45:21.507811 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.507798 5932 scope.go:122] "RemoveContainer" containerID="7d35493e34716dedcf6bb93978f00b92b63f3e1421c3510d6f1e322a8cff1cbb" Nov 03 20:45:21.508013 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:21.507998 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5_odh-ai-gateway-infra(6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" Nov 03 20:45:21.514031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.514013 5932 scope.go:122] "RemoveContainer" containerID="8a4e4315357031e2f9ab3aa11a3015b017e9c1354fe4ccb6e5b3a17443ac209b" Nov 03 20:45:21.526429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.526414 5932 scope.go:122] "RemoveContainer" containerID="efabb5d58a3db9468997785153a0fc983d2b91c589ec4867c24af84183022f6b" Nov 03 20:45:21.607946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.607914 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh"] Nov 03 20:45:21.718476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.718443 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7"] Nov 03 20:45:21.825800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:21.825721 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5"] Nov 03 20:45:22.048311 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.048280 5932 scope.go:122] "RemoveContainer" containerID="7f50d2ca32dfaaf46a718c496bb7340652266663bc75b6d39884a5ff9fb260b2" Nov 03 20:45:22.516408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.516377 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" event={"ID":"cf1c8b6a-6a98-44b6-aae9-13a517e734f5","Type":"ContainerStarted","Data":"dc7f161f7245be8918017a3fe7c021890f5fcfe6fd2b0f215f65bddb21477e17"} Nov 03 20:45:22.519937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.519912 5932 generic.go:356] "Generic (PLEG): container finished" podID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerID="421c575f961602b20e9611fe30e41db7b1aa75a1246b24faac2cd619dbd79c0d" exitCode=60 Nov 03 20:45:22.520061 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.519994 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerDied","Data":"421c575f961602b20e9611fe30e41db7b1aa75a1246b24faac2cd619dbd79c0d"} Nov 03 20:45:22.520101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.520067 5932 scope.go:122] "RemoveContainer" containerID="8e724becbd5ebc55be6a7a716246598f047711d9edcf2a0fd6da81622066871e" Nov 03 20:45:22.520441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.520423 5932 scope.go:122] "RemoveContainer" containerID="421c575f961602b20e9611fe30e41db7b1aa75a1246b24faac2cd619dbd79c0d" Nov 03 20:45:22.520760 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:22.520735 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl_odh-ai-gateway-infra(1b2d030d-70d7-4162-bab8-0f0bec346b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" Nov 03 20:45:22.632085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.631938 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" podStartSLOduration=17.514820048 podStartE2EDuration="22.63191805s" totalImagesPullingTime="5.117098002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:22.624353864 +0000 UTC m=+930.689595333" watchObservedRunningTime="2026-11-03 20:45:22.63191805 +0000 UTC m=+930.697159524" Nov 03 20:45:22.680885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.680858 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:22.731947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.731921 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:22.759085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.759063 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6\" (UniqueName: \"kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6\") pod \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " Nov 03 20:45:22.759221 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.759155 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca\") pod \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\" (UID: \"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de\") " Nov 03 20:45:22.759531 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.759505 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca" pod "d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" (UID: "d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:22.761387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.761360 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6" pod "d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" (UID: "d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de"). InnerVolumeSpecName "kube-api-access-zt4b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:22.763905 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.763888 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:22.859984 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.859956 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58\" (UniqueName: \"kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58\") pod \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " Nov 03 20:45:22.860118 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860031 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca\") pod \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\" (UID: \"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e\") " Nov 03 20:45:22.860195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860121 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca\") pod \"a2b1d05e-b502-47f0-9f50-370d8be91728\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " Nov 03 20:45:22.860195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860156 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc\" (UniqueName: \"kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc\") pod \"a2b1d05e-b502-47f0-9f50-370d8be91728\" (UID: \"a2b1d05e-b502-47f0-9f50-370d8be91728\") " Nov 03 20:45:22.860440 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860413 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca" pod "6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" (UID: "6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:22.860569 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860496 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca" pod "a2b1d05e-b502-47f0-9f50-370d8be91728" (UID: "a2b1d05e-b502-47f0-9f50-370d8be91728"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:22.860660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860637 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4b6\" (UniqueName: \"kubernetes.io/projected/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-kube-api-access-zt4b6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.860725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860660 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b1d05e-b502-47f0-9f50-370d8be91728-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.860725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860676 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.860725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.860691 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.862675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.862647 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58" pod "6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" (UID: "6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e"). InnerVolumeSpecName "kube-api-access-t5z58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:22.863147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.863119 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc" pod "a2b1d05e-b502-47f0-9f50-370d8be91728" (UID: "a2b1d05e-b502-47f0-9f50-370d8be91728"). InnerVolumeSpecName "kube-api-access-6xwfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:22.961700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.961676 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5z58\" (UniqueName: \"kubernetes.io/projected/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e-kube-api-access-t5z58\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.961700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.961700 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwfc\" (UniqueName: \"kubernetes.io/projected/a2b1d05e-b502-47f0-9f50-370d8be91728-kube-api-access-6xwfc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:22.966998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:22.966971 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl"] Nov 03 20:45:23.043981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.043958 5932 scope.go:122] "RemoveContainer" containerID="2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa" Nov 03 20:45:23.531312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.531285 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" Nov 03 20:45:23.531763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.531311 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh" event={"ID":"d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de","Type":"ContainerDied","Data":"668a4e0ae30678024f8ff7075c3eafe2c8ebab259700025dc86e817fb92da690"} Nov 03 20:45:23.531763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.531373 5932 scope.go:122] "RemoveContainer" containerID="65ea0216a1a9eb6dee79b0533a7803b2e540ba21e7c1d3f62b0f02541551a8c4" Nov 03 20:45:23.533586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.533561 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerStarted","Data":"3c3f1850cb3527d074665f6a3a76094ab01b9f97ba221ac584111b1b96c152c8"} Nov 03 20:45:23.537036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.537013 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" Nov 03 20:45:23.537163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.537049 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7" event={"ID":"a2b1d05e-b502-47f0-9f50-370d8be91728","Type":"ContainerDied","Data":"f4fac8675a05fef77292f307c895031b8f884d8a90a3fcb70f7548410f44ba57"} Nov 03 20:45:23.538693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.538671 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" event={"ID":"6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e","Type":"ContainerDied","Data":"b3963177b181b97592562f25874e3b88195873df5ab7cdec62f09555375e9f09"} Nov 03 20:45:23.538857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.538677 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5" Nov 03 20:45:23.541237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.541137 5932 generic.go:356] "Generic (PLEG): container finished" podID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerID="dc7f161f7245be8918017a3fe7c021890f5fcfe6fd2b0f215f65bddb21477e17" exitCode=60 Nov 03 20:45:23.541237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.541172 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" event={"ID":"cf1c8b6a-6a98-44b6-aae9-13a517e734f5","Type":"ContainerDied","Data":"dc7f161f7245be8918017a3fe7c021890f5fcfe6fd2b0f215f65bddb21477e17"} Nov 03 20:45:23.542027 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.541545 5932 scope.go:122] "RemoveContainer" containerID="dc7f161f7245be8918017a3fe7c021890f5fcfe6fd2b0f215f65bddb21477e17" Nov 03 20:45:23.542027 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:23.541899 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k_odh-ai-gateway-infra(cf1c8b6a-6a98-44b6-aae9-13a517e734f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" Nov 03 20:45:23.543329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.543308 5932 scope.go:122] "RemoveContainer" containerID="fa576f4f7ba6abf9593246f838314b3ad7fe3e92ca1346fbc94156f0b0db70db" Nov 03 20:45:23.556306 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.556227 5932 scope.go:122] "RemoveContainer" containerID="7d35493e34716dedcf6bb93978f00b92b63f3e1421c3510d6f1e322a8cff1cbb" Nov 03 20:45:23.598863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.598831 5932 scope.go:122] "RemoveContainer" containerID="7f50d2ca32dfaaf46a718c496bb7340652266663bc75b6d39884a5ff9fb260b2" Nov 03 20:45:23.635269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.635240 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k"] Nov 03 20:45:23.690751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.690724 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh"] Nov 03 20:45:23.693804 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.693782 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895645-fvvgh"] Nov 03 20:45:23.709288 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.709267 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:23.774851 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.774820 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df\" (UniqueName: \"kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df\") pod \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " Nov 03 20:45:23.775024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.774943 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca\") pod \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\" (UID: \"1b2d030d-70d7-4162-bab8-0f0bec346b8a\") " Nov 03 20:45:23.775314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.775291 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca" pod "1b2d030d-70d7-4162-bab8-0f0bec346b8a" (UID: "1b2d030d-70d7-4162-bab8-0f0bec346b8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:23.777025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.777002 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df" pod "1b2d030d-70d7-4162-bab8-0f0bec346b8a" (UID: "1b2d030d-70d7-4162-bab8-0f0bec346b8a"). InnerVolumeSpecName "kube-api-access-8k7df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:23.798972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.798945 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5"] Nov 03 20:45:23.808798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.808765 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895645-ggjm5"] Nov 03 20:45:23.876549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.876516 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7df\" (UniqueName: \"kubernetes.io/projected/1b2d030d-70d7-4162-bab8-0f0bec346b8a-kube-api-access-8k7df\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:23.876702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:23.876551 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b2d030d-70d7-4162-bab8-0f0bec346b8a-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:24.023784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.023751 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7"] Nov 03 20:45:24.029864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.029837 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3d2008-29895645-45sj7"] Nov 03 20:45:24.052355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.052326 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" path="/var/lib/kubelet/pods/6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e/volumes" Nov 03 20:45:24.052750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.052736 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" path="/var/lib/kubelet/pods/a2b1d05e-b502-47f0-9f50-370d8be91728/volumes" Nov 03 20:45:24.053087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.053075 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" path="/var/lib/kubelet/pods/d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de/volumes" Nov 03 20:45:24.550513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.550484 5932 generic.go:356] "Generic (PLEG): container finished" podID="1639bf18-6962-45a1-90b9-00c5431ded13" containerID="3c3f1850cb3527d074665f6a3a76094ab01b9f97ba221ac584111b1b96c152c8" exitCode=60 Nov 03 20:45:24.551019 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.550559 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerDied","Data":"3c3f1850cb3527d074665f6a3a76094ab01b9f97ba221ac584111b1b96c152c8"} Nov 03 20:45:24.551019 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.550648 5932 scope.go:122] "RemoveContainer" containerID="2c8309fe3b6da6ca3bee4f661b4a7810910c6dccfb66ecc5a01f7eaaf0abe8aa" Nov 03 20:45:24.551019 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.550968 5932 scope.go:122] "RemoveContainer" containerID="3c3f1850cb3527d074665f6a3a76094ab01b9f97ba221ac584111b1b96c152c8" Nov 03 20:45:24.551255 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:24.551236 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls_odh-ai-gateway-infra(1639bf18-6962-45a1-90b9-00c5431ded13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" Nov 03 20:45:24.552019 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.551996 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" Nov 03 20:45:24.552082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.552023 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl" event={"ID":"1b2d030d-70d7-4162-bab8-0f0bec346b8a","Type":"ContainerDied","Data":"7e488f792847ec4077f7ef3ff10fd965e1bd2cba5137f53c47dc853c8424cc2c"} Nov 03 20:45:24.564344 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.564327 5932 scope.go:122] "RemoveContainer" containerID="421c575f961602b20e9611fe30e41db7b1aa75a1246b24faac2cd619dbd79c0d" Nov 03 20:45:24.712791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.712763 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:24.785286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.785255 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca\") pod \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " Nov 03 20:45:24.785446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.785427 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5\" (UniqueName: \"kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5\") pod \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\" (UID: \"cf1c8b6a-6a98-44b6-aae9-13a517e734f5\") " Nov 03 20:45:24.785590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.785568 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca" pod "cf1c8b6a-6a98-44b6-aae9-13a517e734f5" (UID: "cf1c8b6a-6a98-44b6-aae9-13a517e734f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:24.785688 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.785675 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:24.787438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.787412 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5" pod "cf1c8b6a-6a98-44b6-aae9-13a517e734f5" (UID: "cf1c8b6a-6a98-44b6-aae9-13a517e734f5"). InnerVolumeSpecName "kube-api-access-mnhc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:24.871444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.871353 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl"] Nov 03 20:45:24.875592 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.875548 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3b8feb-29895645-n8vsl"] Nov 03 20:45:24.886889 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:24.886865 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnhc5\" (UniqueName: \"kubernetes.io/projected/cf1c8b6a-6a98-44b6-aae9-13a517e734f5-kube-api-access-mnhc5\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:25.259341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.259310 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls"] Nov 03 20:45:25.566482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.566453 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" event={"ID":"cf1c8b6a-6a98-44b6-aae9-13a517e734f5","Type":"ContainerDied","Data":"0173d210a873f0f0b3ccac47b07cee8bfae940ae08000166d600f8340bf299d8"} Nov 03 20:45:25.566896 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.566496 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k" Nov 03 20:45:25.566896 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.566514 5932 scope.go:122] "RemoveContainer" containerID="dc7f161f7245be8918017a3fe7c021890f5fcfe6fd2b0f215f65bddb21477e17" Nov 03 20:45:25.710395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.710373 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:25.783801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.783771 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k"] Nov 03 20:45:25.792803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.792771 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895645-42w6k"] Nov 03 20:45:25.797364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.797344 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t\" (UniqueName: \"kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t\") pod \"1639bf18-6962-45a1-90b9-00c5431ded13\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " Nov 03 20:45:25.797436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.797383 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca\") pod \"1639bf18-6962-45a1-90b9-00c5431ded13\" (UID: \"1639bf18-6962-45a1-90b9-00c5431ded13\") " Nov 03 20:45:25.797773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.797753 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca" pod "1639bf18-6962-45a1-90b9-00c5431ded13" (UID: "1639bf18-6962-45a1-90b9-00c5431ded13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:25.799549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.799521 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t" pod "1639bf18-6962-45a1-90b9-00c5431ded13" (UID: "1639bf18-6962-45a1-90b9-00c5431ded13"). InnerVolumeSpecName "kube-api-access-9js8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:25.898598 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.898512 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9js8t\" (UniqueName: \"kubernetes.io/projected/1639bf18-6962-45a1-90b9-00c5431ded13-kube-api-access-9js8t\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:25.898598 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:25.898552 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1639bf18-6962-45a1-90b9-00c5431ded13-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:26.051843 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.051810 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" path="/var/lib/kubelet/pods/1b2d030d-70d7-4162-bab8-0f0bec346b8a/volumes" Nov 03 20:45:26.052195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.052181 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" path="/var/lib/kubelet/pods/cf1c8b6a-6a98-44b6-aae9-13a517e734f5/volumes" Nov 03 20:45:26.575174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.575147 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" Nov 03 20:45:26.575670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.575184 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls" event={"ID":"1639bf18-6962-45a1-90b9-00c5431ded13","Type":"ContainerDied","Data":"9e3354668a3c7b00f1bd09e14c04e612c0de90bfd61bc6cc3dab08fb300649a5"} Nov 03 20:45:26.575670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.575245 5932 scope.go:122] "RemoveContainer" containerID="3c3f1850cb3527d074665f6a3a76094ab01b9f97ba221ac584111b1b96c152c8" Nov 03 20:45:26.730740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.730711 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls"] Nov 03 20:45:26.776689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:26.776667 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895645-qfcls"] Nov 03 20:45:28.052993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:28.052961 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" path="/var/lib/kubelet/pods/1639bf18-6962-45a1-90b9-00c5431ded13/volumes" Nov 03 20:45:37.674919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.674887 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956"] Nov 03 20:45:37.675399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675386 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.675437 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675399 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.675437 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675409 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.675437 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675415 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.675437 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675424 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.675437 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675428 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.675437 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675438 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675442 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675449 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675454 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675464 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675468 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675474 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675478 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675495 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675499 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675508 5932 cpu_manager.go:414] "containerMap: removing container" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675512 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675524 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675528 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675535 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675539 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675547 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675551 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675558 5932 cpu_manager.go:414] "containerMap: removing container" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675562 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675569 5932 cpu_manager.go:414] "containerMap: removing container" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675573 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675587 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.675653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675592 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675680 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675689 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675696 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675704 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675710 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675715 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675721 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675726 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675731 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf1c8b6a-6a98-44b6-aae9-13a517e734f5" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675738 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675743 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675748 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675752 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675756 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:37.676328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.675760 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:37.683089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.683070 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.733962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.733936 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956"] Nov 03 20:45:37.777015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.776979 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:45:37.777154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.777059 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:45:37.831028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.830995 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.831185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.831038 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54dzc\" (UniqueName: \"kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.831185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.831064 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.932406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.932320 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.932406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.932363 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54dzc\" (UniqueName: \"kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.932406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.932390 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.933002 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.932971 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.934725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.934704 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.990155 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.990129 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54dzc\" (UniqueName: \"kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc\") pod \"maas-api-revoke-sub-41b2b3b4-l8956\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:37.996945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:37.996927 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:38.371938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:38.371859 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956"] Nov 03 20:45:38.377785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:38.377759 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36cb217_8eaa_47b8_b1d4_bd975f17a548.slice/crio-a4715b5024c8517a5b1f01582c78b8b1d585d7df6f3bcb63370d65ce4a77e32b WatchSource:0}: Error finding container a4715b5024c8517a5b1f01582c78b8b1d585d7df6f3bcb63370d65ce4a77e32b: Status 404 returned error can't find the container with id a4715b5024c8517a5b1f01582c78b8b1d585d7df6f3bcb63370d65ce4a77e32b Nov 03 20:45:38.661438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:38.661406 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" event={"ID":"d36cb217-8eaa-47b8-b1d4-bd975f17a548","Type":"ContainerStarted","Data":"c51614609c0158a59f83b62b36664afcfacc1afc541ef0581e56da1ce7d43ab9"} Nov 03 20:45:38.661552 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:38.661466 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" event={"ID":"d36cb217-8eaa-47b8-b1d4-bd975f17a548","Type":"ContainerStarted","Data":"a4715b5024c8517a5b1f01582c78b8b1d585d7df6f3bcb63370d65ce4a77e32b"} Nov 03 20:45:38.769943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:38.769878 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" podStartSLOduration=1.7698637069999998 podStartE2EDuration="1.769863707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:38.768852835 +0000 UTC m=+946.834094308" watchObservedRunningTime="2026-11-03 20:45:38.769863707 +0000 UTC m=+946.835105180" Nov 03 20:45:39.669455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:39.669412 5932 generic.go:356] "Generic (PLEG): container finished" podID="d36cb217-8eaa-47b8-b1d4-bd975f17a548" containerID="c51614609c0158a59f83b62b36664afcfacc1afc541ef0581e56da1ce7d43ab9" exitCode=0 Nov 03 20:45:39.669673 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:39.669495 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" event={"ID":"d36cb217-8eaa-47b8-b1d4-bd975f17a548","Type":"ContainerDied","Data":"c51614609c0158a59f83b62b36664afcfacc1afc541ef0581e56da1ce7d43ab9"} Nov 03 20:45:40.826321 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.826297 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:40.965441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.965363 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc\" (UniqueName: \"kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc\") pod \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " Nov 03 20:45:40.965602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.965468 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca\") pod \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " Nov 03 20:45:40.965602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.965526 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca\") pod \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\" (UID: \"d36cb217-8eaa-47b8-b1d4-bd975f17a548\") " Nov 03 20:45:40.966030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.965993 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca" pod "d36cb217-8eaa-47b8-b1d4-bd975f17a548" (UID: "d36cb217-8eaa-47b8-b1d4-bd975f17a548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:40.967674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.967635 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca" pod "d36cb217-8eaa-47b8-b1d4-bd975f17a548" (UID: "d36cb217-8eaa-47b8-b1d4-bd975f17a548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:40.967674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:40.967644 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc" pod "d36cb217-8eaa-47b8-b1d4-bd975f17a548" (UID: "d36cb217-8eaa-47b8-b1d4-bd975f17a548"). InnerVolumeSpecName "kube-api-access-54dzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:41.067839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:41.067787 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54dzc\" (UniqueName: \"kubernetes.io/projected/d36cb217-8eaa-47b8-b1d4-bd975f17a548-kube-api-access-54dzc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:41.069273 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:41.069249 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:41.071595 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:41.071574 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d36cb217-8eaa-47b8-b1d4-bd975f17a548-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:41.689228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:41.689199 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" Nov 03 20:45:41.689228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:41.689207 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956" event={"ID":"d36cb217-8eaa-47b8-b1d4-bd975f17a548","Type":"ContainerDied","Data":"a4715b5024c8517a5b1f01582c78b8b1d585d7df6f3bcb63370d65ce4a77e32b"} Nov 03 20:45:41.689438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:41.689247 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4715b5024c8517a5b1f01582c78b8b1d585d7df6f3bcb63370d65ce4a77e32b" Nov 03 20:45:43.593098 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.593059 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2"] Nov 03 20:45:43.593996 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.593973 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:43.594146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594126 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3b0293-44d4-4b5b-8ad8-1c50ae391b1e" containerName="cleanup" Nov 03 20:45:43.594235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594153 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d36cb217-8eaa-47b8-b1d4-bd975f17a548" containerName="revoke-keys" Nov 03 20:45:43.594235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594161 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d36cb217-8eaa-47b8-b1d4-bd975f17a548" containerName="revoke-keys" Nov 03 20:45:43.594235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594169 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:43.594235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594173 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2b1d05e-b502-47f0-9f50-370d8be91728" containerName="cleanup" Nov 03 20:45:43.594235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594184 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:43.594235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594190 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:43.594418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594305 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d940fd6e-a7fa-43a1-ac47-ca1ebfb1d1de" containerName="cleanup" Nov 03 20:45:43.594418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594322 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1639bf18-6962-45a1-90b9-00c5431ded13" containerName="cleanup" Nov 03 20:45:43.594418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594328 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d36cb217-8eaa-47b8-b1d4-bd975f17a548" containerName="revoke-keys" Nov 03 20:45:43.594418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.594336 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b2d030d-70d7-4162-bab8-0f0bec346b8a" containerName="cleanup" Nov 03 20:45:43.597893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.597877 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.694250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.694221 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"openshift-service-ca.crt\"" Nov 03 20:45:43.694250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.694234 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"premium-w6-504eb0-kserve-self-signed-certs\"" Nov 03 20:45:43.694509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.694267 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"kube-root-ca.crt\"" Nov 03 20:45:43.694509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.694389 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-504eb0\"/\"default-dockercfg-56s89\"" Nov 03 20:45:43.697150 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697125 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2"] Nov 03 20:45:43.697265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697202 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.697265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697230 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.697380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697347 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.697380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697369 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.697455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697410 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.697512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.697477 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78b94\" (UniqueName: \"kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.755811 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.755676 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp"] Nov 03 20:45:43.760683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.760656 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.798197 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798166 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798204 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798271 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798293 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78b94\" (UniqueName: \"kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798329 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.798486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798413 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798482 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmd4g\" (UniqueName: \"kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.798601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798540 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798608 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.798731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798725 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798844 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798788 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.798893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.798862 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.800617 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.800596 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.800744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.800734 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.804694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.804655 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp"] Nov 03 20:45:43.810400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.810373 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:45:43.810521 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.810442 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:45:43.856156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.856098 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78b94\" (UniqueName: \"kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94\") pod \"premium-w6-504eb0-kserve-6845dbfd56-hcsc2\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.899334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.899308 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.899482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.899399 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmd4g\" (UniqueName: \"kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.899482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.899437 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.900035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.900012 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.901469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.901446 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:43.910138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.910118 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:43.957141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:43.957110 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmd4g\" (UniqueName: \"kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g\") pod \"maas-api-revoke-sub-7d747c76-ndpxp\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:44.074017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.073983 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:44.159121 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:45:44.159079 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:45:44.283000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.282967 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2"] Nov 03 20:45:44.288682 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:44.288657 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e7c8a5_e276_418a_b4d5_474923fbff63.slice/crio-416754cec034ed7665c725a08003e8baac7d766a24fd37b10a596b28aa172636 WatchSource:0}: Error finding container 416754cec034ed7665c725a08003e8baac7d766a24fd37b10a596b28aa172636: Status 404 returned error can't find the container with id 416754cec034ed7665c725a08003e8baac7d766a24fd37b10a596b28aa172636 Nov 03 20:45:44.445890 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.445858 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp"] Nov 03 20:45:44.451826 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:44.451802 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e22d4c_2e1f_494f_96c2_984353adbaf5.slice/crio-841631d801ca23f372f55b89d56df1e276bdfd492f64be170700c1845cabe5c9 WatchSource:0}: Error finding container 841631d801ca23f372f55b89d56df1e276bdfd492f64be170700c1845cabe5c9: Status 404 returned error can't find the container with id 841631d801ca23f372f55b89d56df1e276bdfd492f64be170700c1845cabe5c9 Nov 03 20:45:44.709712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.709644 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" event={"ID":"87e22d4c-2e1f-494f-96c2-984353adbaf5","Type":"ContainerStarted","Data":"841631d801ca23f372f55b89d56df1e276bdfd492f64be170700c1845cabe5c9"} Nov 03 20:45:44.711488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.711463 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" event={"ID":"09e7c8a5-e276-418a-b4d5-474923fbff63","Type":"ContainerStarted","Data":"bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e"} Nov 03 20:45:44.711603 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.711495 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" event={"ID":"09e7c8a5-e276-418a-b4d5-474923fbff63","Type":"ContainerStarted","Data":"416754cec034ed7665c725a08003e8baac7d766a24fd37b10a596b28aa172636"} Nov 03 20:45:44.822216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.822163 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podStartSLOduration=1.822149746 podStartE2EDuration="1.822149746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:44.820302147 +0000 UTC m=+952.885543622" watchObservedRunningTime="2026-11-03 20:45:44.822149746 +0000 UTC m=+952.887391218" Nov 03 20:45:44.910800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:44.910772 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:45.719920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:45.719882 5932 generic.go:356] "Generic (PLEG): container finished" podID="87e22d4c-2e1f-494f-96c2-984353adbaf5" containerID="09c0d2177a88b35a4cf4654ff12dda9d410a7c5b51944b3cebc7c78a65544ca9" exitCode=0 Nov 03 20:45:45.720353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:45.719962 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" event={"ID":"87e22d4c-2e1f-494f-96c2-984353adbaf5","Type":"ContainerDied","Data":"09c0d2177a88b35a4cf4654ff12dda9d410a7c5b51944b3cebc7c78a65544ca9"} Nov 03 20:45:46.872517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.872493 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:46.929439 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.929414 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g\" (UniqueName: \"kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g\") pod \"87e22d4c-2e1f-494f-96c2-984353adbaf5\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " Nov 03 20:45:46.929570 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.929516 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca\") pod \"87e22d4c-2e1f-494f-96c2-984353adbaf5\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " Nov 03 20:45:46.929570 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.929548 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca\") pod \"87e22d4c-2e1f-494f-96c2-984353adbaf5\" (UID: \"87e22d4c-2e1f-494f-96c2-984353adbaf5\") " Nov 03 20:45:46.929976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.929947 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca" pod "87e22d4c-2e1f-494f-96c2-984353adbaf5" (UID: "87e22d4c-2e1f-494f-96c2-984353adbaf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:46.931438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.931418 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g" pod "87e22d4c-2e1f-494f-96c2-984353adbaf5" (UID: "87e22d4c-2e1f-494f-96c2-984353adbaf5"). InnerVolumeSpecName "kube-api-access-bmd4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:46.931503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:46.931442 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca" pod "87e22d4c-2e1f-494f-96c2-984353adbaf5" (UID: "87e22d4c-2e1f-494f-96c2-984353adbaf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:47.031303 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:47.031209 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:47.031303 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:47.031241 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87e22d4c-2e1f-494f-96c2-984353adbaf5-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:47.031303 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:47.031251 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmd4g\" (UniqueName: \"kubernetes.io/projected/87e22d4c-2e1f-494f-96c2-984353adbaf5-kube-api-access-bmd4g\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:47.733386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:47.733358 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" Nov 03 20:45:47.733547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:47.733361 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp" event={"ID":"87e22d4c-2e1f-494f-96c2-984353adbaf5","Type":"ContainerDied","Data":"841631d801ca23f372f55b89d56df1e276bdfd492f64be170700c1845cabe5c9"} Nov 03 20:45:47.733547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:47.733482 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="841631d801ca23f372f55b89d56df1e276bdfd492f64be170700c1845cabe5c9" Nov 03 20:45:48.430703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.430675 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5"] Nov 03 20:45:48.431417 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.431393 5932 cpu_manager.go:414] "containerMap: removing container" podUID="87e22d4c-2e1f-494f-96c2-984353adbaf5" containerName="revoke-keys" Nov 03 20:45:48.431516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.431418 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e22d4c-2e1f-494f-96c2-984353adbaf5" containerName="revoke-keys" Nov 03 20:45:48.431675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.431651 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e22d4c-2e1f-494f-96c2-984353adbaf5" containerName="revoke-keys" Nov 03 20:45:48.435004 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.434984 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.531035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.531009 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:45:48.531035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.531004 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:45:48.534015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.533989 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5"] Nov 03 20:45:48.548284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.548254 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6b6\" (UniqueName: \"kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.548561 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.548542 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.548772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.548753 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.650054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.650020 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.650241 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.650152 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6b6\" (UniqueName: \"kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.650241 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.650213 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.650941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.650918 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.652642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.652595 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.707748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.707674 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6b6\" (UniqueName: \"kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6\") pod \"maas-api-revoke-sub-c6158410-wrlc5\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:48.748681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:48.748649 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:49.125859 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:49.125827 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5"] Nov 03 20:45:49.132052 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:45:49.132027 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f2c3c9_d951_4423_8e0c_f63266099584.slice/crio-8e84e53c70a4c64ddd958113163020f4f6a8301abecba2bac7b9b264fb1f457b WatchSource:0}: Error finding container 8e84e53c70a4c64ddd958113163020f4f6a8301abecba2bac7b9b264fb1f457b: Status 404 returned error can't find the container with id 8e84e53c70a4c64ddd958113163020f4f6a8301abecba2bac7b9b264fb1f457b Nov 03 20:45:49.751666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:49.751565 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" event={"ID":"71f2c3c9-d951-4423-8e0c-f63266099584","Type":"ContainerStarted","Data":"5481827a24e3fb15a7386326168d30b42bb7c98e1c8832218df69d91da8059e5"} Nov 03 20:45:49.751666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:49.751614 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" event={"ID":"71f2c3c9-d951-4423-8e0c-f63266099584","Type":"ContainerStarted","Data":"8e84e53c70a4c64ddd958113163020f4f6a8301abecba2bac7b9b264fb1f457b"} Nov 03 20:45:49.862236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:49.862185 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" podStartSLOduration=1.862169969 podStartE2EDuration="1.862169969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:45:49.859417265 +0000 UTC m=+957.924658737" watchObservedRunningTime="2026-11-03 20:45:49.862169969 +0000 UTC m=+957.927411442" Nov 03 20:45:50.759935 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:50.759896 5932 generic.go:356] "Generic (PLEG): container finished" podID="71f2c3c9-d951-4423-8e0c-f63266099584" containerID="5481827a24e3fb15a7386326168d30b42bb7c98e1c8832218df69d91da8059e5" exitCode=0 Nov 03 20:45:50.760391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:50.759976 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" event={"ID":"71f2c3c9-d951-4423-8e0c-f63266099584","Type":"ContainerDied","Data":"5481827a24e3fb15a7386326168d30b42bb7c98e1c8832218df69d91da8059e5"} Nov 03 20:45:51.918530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.918505 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:51.988051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.988018 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca\") pod \"71f2c3c9-d951-4423-8e0c-f63266099584\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " Nov 03 20:45:51.988219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.988059 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6\" (UniqueName: \"kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6\") pod \"71f2c3c9-d951-4423-8e0c-f63266099584\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " Nov 03 20:45:51.988219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.988162 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca\") pod \"71f2c3c9-d951-4423-8e0c-f63266099584\" (UID: \"71f2c3c9-d951-4423-8e0c-f63266099584\") " Nov 03 20:45:51.988521 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.988497 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca" pod "71f2c3c9-d951-4423-8e0c-f63266099584" (UID: "71f2c3c9-d951-4423-8e0c-f63266099584"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:45:51.990215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.990194 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca" pod "71f2c3c9-d951-4423-8e0c-f63266099584" (UID: "71f2c3c9-d951-4423-8e0c-f63266099584"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:45:51.990317 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:51.990293 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6" pod "71f2c3c9-d951-4423-8e0c-f63266099584" (UID: "71f2c3c9-d951-4423-8e0c-f63266099584"). InnerVolumeSpecName "kube-api-access-5n6b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:45:52.089602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:52.089570 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:52.089602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:52.089602 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n6b6\" (UniqueName: \"kubernetes.io/projected/71f2c3c9-d951-4423-8e0c-f63266099584-kube-api-access-5n6b6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:52.089797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:52.089633 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f2c3c9-d951-4423-8e0c-f63266099584-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:45:52.774027 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:52.773992 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" Nov 03 20:45:52.774027 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:52.774001 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5" event={"ID":"71f2c3c9-d951-4423-8e0c-f63266099584","Type":"ContainerDied","Data":"8e84e53c70a4c64ddd958113163020f4f6a8301abecba2bac7b9b264fb1f457b"} Nov 03 20:45:52.774233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:52.774073 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e84e53c70a4c64ddd958113163020f4f6a8301abecba2bac7b9b264fb1f457b" Nov 03 20:45:53.911212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:53.911174 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:53.924075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:53.924049 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:45:54.797886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:45:54.797858 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:46:08.221325 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.221283 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n"] Nov 03 20:46:08.222217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.222196 5932 cpu_manager.go:414] "containerMap: removing container" podUID="71f2c3c9-d951-4423-8e0c-f63266099584" containerName="revoke-keys" Nov 03 20:46:08.222217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.222217 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f2c3c9-d951-4423-8e0c-f63266099584" containerName="revoke-keys" Nov 03 20:46:08.222397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.222365 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f2c3c9-d951-4423-8e0c-f63266099584" containerName="revoke-keys" Nov 03 20:46:08.225830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.225806 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.290552 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.290523 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1e6fe1-openshift-default-dockercfg-gpj7x\"" Nov 03 20:46:08.355435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.355402 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n"] Nov 03 20:46:08.383770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.383740 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.383940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.383790 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.383940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.383881 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.383940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.383910 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.383940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.383931 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68gt\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.384123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.383966 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.384123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.384115 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.384219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.384182 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.384303 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.384289 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.485773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485697 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.485773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485738 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.485773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485754 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.485773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485773 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68gt\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485807 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485840 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485891 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.485956 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.486003 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.486238 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.486251 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.486472 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.486686 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.486834 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.486748 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.488320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.488298 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.488852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.488831 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.543194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.543169 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68gt\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.547122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.547099 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:08.839354 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:08.839280 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:09.270848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.270816 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n"] Nov 03 20:46:09.276724 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:09.276697 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff55790_9ea4_4411_a3b7_6e1c5eabe989.slice/crio-baec34eb639dd38a3eb722d3c6470eeae8cb95f7f6efa4caa7da25d1e2c9c3d9 WatchSource:0}: Error finding container baec34eb639dd38a3eb722d3c6470eeae8cb95f7f6efa4caa7da25d1e2c9c3d9: Status 404 returned error can't find the container with id baec34eb639dd38a3eb722d3c6470eeae8cb95f7f6efa4caa7da25d1e2c9c3d9 Nov 03 20:46:09.279113 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.279078 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:09.279197 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.279146 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:09.279241 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.279196 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:09.722752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.722724 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk"] Nov 03 20:46:09.726861 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.726844 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.821753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.821722 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2a5ec9-openshift-default-dockercfg-kln2c\"" Nov 03 20:46:09.824569 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.824548 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk"] Nov 03 20:46:09.889008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.888974 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" event={"ID":"fff55790-9ea4-4411-a3b7-6e1c5eabe989","Type":"ContainerStarted","Data":"b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0"} Nov 03 20:46:09.889146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.889033 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" event={"ID":"fff55790-9ea4-4411-a3b7-6e1c5eabe989","Type":"ContainerStarted","Data":"baec34eb639dd38a3eb722d3c6470eeae8cb95f7f6efa4caa7da25d1e2c9c3d9"} Nov 03 20:46:09.902739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.902717 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.902805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.902782 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.902848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.902829 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.902907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.902887 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.902943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.902930 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hstqb\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.902977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.902951 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.903042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.903029 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.903121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.903108 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:09.903156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:09.903150 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005103 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005023 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005242 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005125 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005242 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005186 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005242 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005223 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005401 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005270 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hstqb\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005401 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005307 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005416 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005477 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.005600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.005562 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.006085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.006058 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.006341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.006320 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.007206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.007178 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.007461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.007440 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.008737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.008678 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podStartSLOduration=2.008660938 podStartE2EDuration="2.008660938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:10.001571359 +0000 UTC m=+978.066812851" watchObservedRunningTime="2026-11-03 20:46:10.008660938 +0000 UTC m=+978.073902413" Nov 03 20:46:10.016463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.012996 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.016463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.013548 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.016975 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.016952 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.065640 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.065595 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hstqb\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.069180 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.069145 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.345985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.345887 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:10.846667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.842003 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:10.846953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.846916 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:46:10.847099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.846971 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:46:10.847243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.847213 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk"] Nov 03 20:46:10.851839 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:10.851810 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17462258_c58b_4211_8cdb_20283a6f1d61.slice/crio-165aed23f0b5fbbadce14ecd21ac33dfdf7578f4d2345fcc8d837e0e3774eb70 WatchSource:0}: Error finding container 165aed23f0b5fbbadce14ecd21ac33dfdf7578f4d2345fcc8d837e0e3774eb70: Status 404 returned error can't find the container with id 165aed23f0b5fbbadce14ecd21ac33dfdf7578f4d2345fcc8d837e0e3774eb70 Nov 03 20:46:10.854266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.854234 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:10.854350 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.854303 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:10.854413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.854357 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:10.897551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:10.897524 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" event={"ID":"17462258-c58b-4211-8cdb-20283a6f1d61","Type":"ContainerStarted","Data":"165aed23f0b5fbbadce14ecd21ac33dfdf7578f4d2345fcc8d837e0e3774eb70"} Nov 03 20:46:11.854646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:11.847846 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:46:11.854646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:11.847908 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:46:11.905673 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:11.905618 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" event={"ID":"17462258-c58b-4211-8cdb-20283a6f1d61","Type":"ContainerStarted","Data":"d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968"} Nov 03 20:46:12.115944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:12.115835 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podStartSLOduration=3.115821978 podStartE2EDuration="3.115821978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:12.06903727 +0000 UTC m=+980.134278744" watchObservedRunningTime="2026-11-03 20:46:12.115821978 +0000 UTC m=+980.181063452" Nov 03 20:46:12.350050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:12.349972 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:12.355648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:12.352583 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:46:12.355648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:12.352672 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 20:46:12.845643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:12.842335 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:46:12.845643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:12.842405 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:46:13.361648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:13.353012 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:46:13.361648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:13.353075 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 20:46:13.846647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:13.845722 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:46:13.846647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:13.845796 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:46:14.194615 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:46:14.194571 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:46:14.361647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:14.361305 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:46:14.361647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:14.361364 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 20:46:14.860648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:14.849040 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:46:14.860648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:14.849104 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:46:15.357646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:15.351970 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:46:15.357646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:15.352029 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 20:46:15.842049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:15.842011 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:46:15.842657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:15.842601 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 20:46:16.355647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:16.350908 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:46:16.355647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:16.350979 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 20:46:17.334837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:17.334736 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:17.335375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:17.335348 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:17.336719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:17.336683 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:46:17.346380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:17.346349 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:46:17.346503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:17.346409 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 20:46:18.587805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:18.587773 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:18.588218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:18.588074 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:18.589327 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:18.589303 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 20:46:19.068388 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.068357 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll"] Nov 03 20:46:19.072716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.072697 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.114285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.114257 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.114462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.114440 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwmd\" (UniqueName: \"kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.114769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.114738 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.124772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.124752 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:46:19.125040 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.125024 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:46:19.131265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.131241 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll"] Nov 03 20:46:19.216024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.215993 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.216213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.216044 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwmd\" (UniqueName: \"kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.216213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.216110 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.216717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.216692 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.218400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.218377 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.273956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.273928 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwmd\" (UniqueName: \"kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd\") pod \"maas-api-revoke-sub-ad45e676-xf6ll\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.385962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.385941 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:19.763804 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:19.763723 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll"] Nov 03 20:46:19.768853 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:19.768824 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6258d6fc_190f_47a2_a8b8_89060f4fe6c5.slice/crio-8988fcbe7e54a10c89a5b06c6cefc7b19f3aaa6d139e2bb57c364e7de4f7963c WatchSource:0}: Error finding container 8988fcbe7e54a10c89a5b06c6cefc7b19f3aaa6d139e2bb57c364e7de4f7963c: Status 404 returned error can't find the container with id 8988fcbe7e54a10c89a5b06c6cefc7b19f3aaa6d139e2bb57c364e7de4f7963c Nov 03 20:46:20.016397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:20.016326 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" event={"ID":"6258d6fc-190f-47a2-a8b8-89060f4fe6c5","Type":"ContainerStarted","Data":"8988fcbe7e54a10c89a5b06c6cefc7b19f3aaa6d139e2bb57c364e7de4f7963c"} Nov 03 20:46:21.025107 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:21.025068 5932 generic.go:356] "Generic (PLEG): container finished" podID="6258d6fc-190f-47a2-a8b8-89060f4fe6c5" containerID="0ff6ef3c663b5930ca538282238a5131bcf5f74115fe63ea6f8e355faa6663a0" exitCode=0 Nov 03 20:46:21.025484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:21.025148 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" event={"ID":"6258d6fc-190f-47a2-a8b8-89060f4fe6c5","Type":"ContainerDied","Data":"0ff6ef3c663b5930ca538282238a5131bcf5f74115fe63ea6f8e355faa6663a0"} Nov 03 20:46:22.187133 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.187107 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:22.250152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.250127 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca\") pod \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " Nov 03 20:46:22.250324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.250174 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd\" (UniqueName: \"kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd\") pod \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " Nov 03 20:46:22.250324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.250230 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca\") pod \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\" (UID: \"6258d6fc-190f-47a2-a8b8-89060f4fe6c5\") " Nov 03 20:46:22.250667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.250638 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca" pod "6258d6fc-190f-47a2-a8b8-89060f4fe6c5" (UID: "6258d6fc-190f-47a2-a8b8-89060f4fe6c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:22.252075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.252043 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca" pod "6258d6fc-190f-47a2-a8b8-89060f4fe6c5" (UID: "6258d6fc-190f-47a2-a8b8-89060f4fe6c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:22.252171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.252129 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd" pod "6258d6fc-190f-47a2-a8b8-89060f4fe6c5" (UID: "6258d6fc-190f-47a2-a8b8-89060f4fe6c5"). InnerVolumeSpecName "kube-api-access-fpwmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:22.351512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.351433 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:22.351512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.351456 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpwmd\" (UniqueName: \"kubernetes.io/projected/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-kube-api-access-fpwmd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:22.351512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:22.351466 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6258d6fc-190f-47a2-a8b8-89060f4fe6c5-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:23.041791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:23.041765 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" Nov 03 20:46:23.041981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:23.041799 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll" event={"ID":"6258d6fc-190f-47a2-a8b8-89060f4fe6c5","Type":"ContainerDied","Data":"8988fcbe7e54a10c89a5b06c6cefc7b19f3aaa6d139e2bb57c364e7de4f7963c"} Nov 03 20:46:23.041981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:23.041846 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8988fcbe7e54a10c89a5b06c6cefc7b19f3aaa6d139e2bb57c364e7de4f7963c" Nov 03 20:46:24.529139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.529106 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl"] Nov 03 20:46:24.529702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.529686 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6258d6fc-190f-47a2-a8b8-89060f4fe6c5" containerName="revoke-keys" Nov 03 20:46:24.529753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.529703 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6258d6fc-190f-47a2-a8b8-89060f4fe6c5" containerName="revoke-keys" Nov 03 20:46:24.529800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.529792 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6258d6fc-190f-47a2-a8b8-89060f4fe6c5" containerName="revoke-keys" Nov 03 20:46:24.533516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.533496 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.577520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577484 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.577726 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577615 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.577726 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577682 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ckq\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.577825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577797 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.577866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577832 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.577866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577853 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.578016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577900 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.578016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.577958 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.578097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.578043 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.582050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.582029 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-504eb0-openshift-default-dockercfg-4ckhb\"" Nov 03 20:46:24.632109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.632077 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl"] Nov 03 20:46:24.678967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.678932 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.678999 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679067 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679182 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ckq\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679260 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679302 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679328 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679392 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679409 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679416 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679465 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679782 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679766 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679849 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.679977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.679903 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.681641 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.681609 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.681880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.681861 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.736504 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.736474 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.736659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.736639 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ckq\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:24.847461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:24.847380 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:25.098840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.098759 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn"] Nov 03 20:46:25.103975 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.103954 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.153070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.152919 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:46:25.153070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.152923 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:46:25.156611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.156587 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn"] Nov 03 20:46:25.186217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.186195 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.186342 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.186240 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c47k\" (UniqueName: \"kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.186342 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.186322 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.275492 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.275466 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl"] Nov 03 20:46:25.280414 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:25.280389 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bec572b_4561_494a_abdc_11a7e4b2f7d3.slice/crio-63ba695460e54495e2e99ae10bd82ca51b22a4564c05865dc100e827a9dd845d WatchSource:0}: Error finding container 63ba695460e54495e2e99ae10bd82ca51b22a4564c05865dc100e827a9dd845d: Status 404 returned error can't find the container with id 63ba695460e54495e2e99ae10bd82ca51b22a4564c05865dc100e827a9dd845d Nov 03 20:46:25.282814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.282783 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:25.282906 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.282844 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:25.282906 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.282873 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:46:25.287179 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.287155 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c47k\" (UniqueName: \"kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.287261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.287206 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.287374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.287355 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.288124 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.288101 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.289450 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.289431 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.344090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.344070 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c47k\" (UniqueName: \"kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k\") pod \"maas-api-revoke-sub-237bfda3-gkwfn\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.418393 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.418357 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:25.794571 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:25.794479 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn"] Nov 03 20:46:25.799980 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:25.799955 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b380ef6_c12b_44a5_b3f1_cf621c7d43ed.slice/crio-a0af1c7a7524afe965748b6ec4177ac6cea9131c92028f2cbf2376f28ea95897 WatchSource:0}: Error finding container a0af1c7a7524afe965748b6ec4177ac6cea9131c92028f2cbf2376f28ea95897: Status 404 returned error can't find the container with id a0af1c7a7524afe965748b6ec4177ac6cea9131c92028f2cbf2376f28ea95897 Nov 03 20:46:26.064811 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.064776 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" event={"ID":"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed","Type":"ContainerStarted","Data":"a0af1c7a7524afe965748b6ec4177ac6cea9131c92028f2cbf2376f28ea95897"} Nov 03 20:46:26.066240 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.066220 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" event={"ID":"3bec572b-4561-494a-abdc-11a7e4b2f7d3","Type":"ContainerStarted","Data":"93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103"} Nov 03 20:46:26.066346 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.066255 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" event={"ID":"3bec572b-4561-494a-abdc-11a7e4b2f7d3","Type":"ContainerStarted","Data":"63ba695460e54495e2e99ae10bd82ca51b22a4564c05865dc100e827a9dd845d"} Nov 03 20:46:26.249073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.248998 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podStartSLOduration=2.248977789 podStartE2EDuration="2.248977789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:26.247158477 +0000 UTC m=+994.312399962" watchObservedRunningTime="2026-11-03 20:46:26.248977789 +0000 UTC m=+994.314219265" Nov 03 20:46:26.848000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.847967 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:26.849661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.849612 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:46:26.849801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:26.849679 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:46:27.074149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:27.074117 5932 generic.go:356] "Generic (PLEG): container finished" podID="6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" containerID="88e0f34eba9cdad3b53270f584f6bd636692d46caf6c93ca878bb401b67cfeb4" exitCode=0 Nov 03 20:46:27.074349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:27.074209 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" event={"ID":"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed","Type":"ContainerDied","Data":"88e0f34eba9cdad3b53270f584f6bd636692d46caf6c93ca878bb401b67cfeb4"} Nov 03 20:46:27.848206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:27.848165 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:46:27.848710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:27.848227 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:46:28.277339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.277310 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:28.356271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.356237 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca\") pod \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " Nov 03 20:46:28.356463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.356310 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k\" (UniqueName: \"kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k\") pod \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " Nov 03 20:46:28.356463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.356368 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca\") pod \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\" (UID: \"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed\") " Nov 03 20:46:28.356799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.356773 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca" pod "6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" (UID: "6b380ef6-c12b-44a5-b3f1-cf621c7d43ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:28.356903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.356874 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:28.358423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.358396 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k" pod "6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" (UID: "6b380ef6-c12b-44a5-b3f1-cf621c7d43ed"). InnerVolumeSpecName "kube-api-access-8c47k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:28.358512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.358456 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca" pod "6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" (UID: "6b380ef6-c12b-44a5-b3f1-cf621c7d43ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:28.458469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.458386 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:28.458469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.458415 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c47k\" (UniqueName: \"kubernetes.io/projected/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed-kube-api-access-8c47k\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:28.848208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.848121 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:46:28.848208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:28.848180 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:46:29.088670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:29.088635 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" Nov 03 20:46:29.088670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:29.088655 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn" event={"ID":"6b380ef6-c12b-44a5-b3f1-cf621c7d43ed","Type":"ContainerDied","Data":"a0af1c7a7524afe965748b6ec4177ac6cea9131c92028f2cbf2376f28ea95897"} Nov 03 20:46:29.088914 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:29.088704 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0af1c7a7524afe965748b6ec4177ac6cea9131c92028f2cbf2376f28ea95897" Nov 03 20:46:29.848101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:29.848069 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:46:29.848289 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:29.848119 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:46:30.848161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:30.848124 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:46:30.848559 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:30.848180 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 20:46:31.996052 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:31.996016 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:31.996526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:31.996472 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:31.998243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:31.998216 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 20:46:42.621375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.621296 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2"] Nov 03 20:46:42.622064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.622044 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" containerName="revoke-keys" Nov 03 20:46:42.622108 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.622065 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" containerName="revoke-keys" Nov 03 20:46:42.622175 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.622166 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" containerName="revoke-keys" Nov 03 20:46:42.625743 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.625723 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.712533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.712502 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.712720 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.712700 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.712777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.712732 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74n4n\" (UniqueName: \"kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.720857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.720827 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:46:42.720987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.720912 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:46:42.724046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.724024 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2"] Nov 03 20:46:42.814251 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.814223 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.814395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.814324 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.814395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.814351 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74n4n\" (UniqueName: \"kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.815021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.815000 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.816686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.816667 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.871442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.871384 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74n4n\" (UniqueName: \"kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n\") pod \"maas-api-revoke-sub-cde5cf20-7fqj2\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:42.939611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:42.939587 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:43.167014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.166973 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg"] Nov 03 20:46:43.171406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.171386 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.219580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.219553 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.219742 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.219589 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6gvq\" (UniqueName: \"kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.219801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.219737 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.228378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.228348 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg"] Nov 03 20:46:43.320902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.320870 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.320902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.320904 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6gvq\" (UniqueName: \"kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.321156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.320967 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.321573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.321546 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.323456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.323434 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.339818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.339794 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2"] Nov 03 20:46:43.346108 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:43.346080 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ba66b4_4f18_4c9f_9cd9_99dc282051cb.slice/crio-aea9ae549981e69cddd745bdd9dad35702644ec68194af2365b5efe8ea935a1e WatchSource:0}: Error finding container aea9ae549981e69cddd745bdd9dad35702644ec68194af2365b5efe8ea935a1e: Status 404 returned error can't find the container with id aea9ae549981e69cddd745bdd9dad35702644ec68194af2365b5efe8ea935a1e Nov 03 20:46:43.378274 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.378249 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6gvq\" (UniqueName: \"kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq\") pod \"maas-api-revoke-sub-0e60940c-kg9zg\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.496708 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.495507 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:43.901653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:43.900028 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg"] Nov 03 20:46:44.195714 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:46:44.194188 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:46:44.195714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:44.195334 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" event={"ID":"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb","Type":"ContainerStarted","Data":"1d9550e87e4ba7a35d4c60ca426df0c55964a68c0904f94c4f781cf21f2f9b25"} Nov 03 20:46:44.195714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:44.195387 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" event={"ID":"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb","Type":"ContainerStarted","Data":"aea9ae549981e69cddd745bdd9dad35702644ec68194af2365b5efe8ea935a1e"} Nov 03 20:46:44.197318 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:44.197279 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" event={"ID":"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc","Type":"ContainerStarted","Data":"c5adb98ffa038446bbb7b35198ab70ba11aca2cf4c6d1bde0bae6bb8e2a43334"} Nov 03 20:46:44.316659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:44.316210 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" podStartSLOduration=2.316189176 podStartE2EDuration="2.316189176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:44.31483548 +0000 UTC m=+1012.380076954" watchObservedRunningTime="2026-11-03 20:46:44.316189176 +0000 UTC m=+1012.381430649" Nov 03 20:46:45.205557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:45.205520 5932 generic.go:356] "Generic (PLEG): container finished" podID="d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" containerID="1d9550e87e4ba7a35d4c60ca426df0c55964a68c0904f94c4f781cf21f2f9b25" exitCode=0 Nov 03 20:46:45.205957 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:45.205612 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" event={"ID":"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb","Type":"ContainerDied","Data":"1d9550e87e4ba7a35d4c60ca426df0c55964a68c0904f94c4f781cf21f2f9b25"} Nov 03 20:46:45.207236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:45.207217 5932 generic.go:356] "Generic (PLEG): container finished" podID="d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" containerID="024913e2c361b962bb09a1c39cf563185a0dc941bb5afb45750acfc5664c0d2e" exitCode=0 Nov 03 20:46:45.207352 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:45.207298 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" event={"ID":"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc","Type":"ContainerDied","Data":"024913e2c361b962bb09a1c39cf563185a0dc941bb5afb45750acfc5664c0d2e"} Nov 03 20:46:46.016950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.016902 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx"] Nov 03 20:46:46.021000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.020981 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.064568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.064543 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.064720 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.064571 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.064720 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.064612 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skl68\" (UniqueName: \"kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.075297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.075273 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx"] Nov 03 20:46:46.165824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.165796 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skl68\" (UniqueName: \"kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.166090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.166069 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.166189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.166128 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.166743 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.166718 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.168405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.168386 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.226946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.226919 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skl68\" (UniqueName: \"kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68\") pod \"maas-api-revoke-sub-8f2f6d61-pp4wx\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.334959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.334822 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:46.377173 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.377141 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:46.406690 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.406659 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:46.472196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472168 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq\" (UniqueName: \"kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq\") pod \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " Nov 03 20:46:46.472359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472281 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca\") pod \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " Nov 03 20:46:46.472359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472308 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca\") pod \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " Nov 03 20:46:46.472359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472349 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n\" (UniqueName: \"kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n\") pod \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " Nov 03 20:46:46.472538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472405 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca\") pod \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\" (UID: \"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb\") " Nov 03 20:46:46.472538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472459 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca\") pod \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\" (UID: \"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc\") " Nov 03 20:46:46.472883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472844 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca" pod "d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" (UID: "d8ba66b4-4f18-4c9f-9cd9-99dc282051cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:46.473036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.472950 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:46.473195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.473162 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca" pod "d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" (UID: "d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:46.475045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.474900 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca" pod "d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" (UID: "d8ba66b4-4f18-4c9f-9cd9-99dc282051cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:46.475045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.474913 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq" pod "d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" (UID: "d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc"). InnerVolumeSpecName "kube-api-access-t6gvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:46.475045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.474960 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca" pod "d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" (UID: "d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:46.475615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.475490 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n" pod "d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" (UID: "d8ba66b4-4f18-4c9f-9cd9-99dc282051cb"). InnerVolumeSpecName "kube-api-access-74n4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:46.573714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.573585 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:46.573714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.573640 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74n4n\" (UniqueName: \"kubernetes.io/projected/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-kube-api-access-74n4n\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:46.573714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.573656 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:46.573714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.573669 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:46.573714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.573682 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6gvq\" (UniqueName: \"kubernetes.io/projected/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc-kube-api-access-t6gvq\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:46.741857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:46.741822 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx"] Nov 03 20:46:46.746526 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:46.746498 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca762bf_f901_433d_9778_8f9ab063db33.slice/crio-4189533107a75105728447dbabdd7762fbf4edde89990b33a0f89edd1abac9b8 WatchSource:0}: Error finding container 4189533107a75105728447dbabdd7762fbf4edde89990b33a0f89edd1abac9b8: Status 404 returned error can't find the container with id 4189533107a75105728447dbabdd7762fbf4edde89990b33a0f89edd1abac9b8 Nov 03 20:46:47.222113 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.222034 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" Nov 03 20:46:47.222113 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.222081 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2" event={"ID":"d8ba66b4-4f18-4c9f-9cd9-99dc282051cb","Type":"ContainerDied","Data":"aea9ae549981e69cddd745bdd9dad35702644ec68194af2365b5efe8ea935a1e"} Nov 03 20:46:47.222359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.222137 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea9ae549981e69cddd745bdd9dad35702644ec68194af2365b5efe8ea935a1e" Nov 03 20:46:47.223537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.223522 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" Nov 03 20:46:47.223739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.223582 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg" event={"ID":"d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc","Type":"ContainerDied","Data":"c5adb98ffa038446bbb7b35198ab70ba11aca2cf4c6d1bde0bae6bb8e2a43334"} Nov 03 20:46:47.223739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.223655 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5adb98ffa038446bbb7b35198ab70ba11aca2cf4c6d1bde0bae6bb8e2a43334" Nov 03 20:46:47.225772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.225746 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" event={"ID":"9ca762bf-f901-433d-9778-8f9ab063db33","Type":"ContainerStarted","Data":"425466fc2e646103fdf54872c4e31fd7bed335b6066f08740c07e6b0d0e97137"} Nov 03 20:46:47.225875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.225782 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" event={"ID":"9ca762bf-f901-433d-9778-8f9ab063db33","Type":"ContainerStarted","Data":"4189533107a75105728447dbabdd7762fbf4edde89990b33a0f89edd1abac9b8"} Nov 03 20:46:47.334700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:47.334646 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" podStartSLOduration=2.334611989 podStartE2EDuration="2.334611989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:47.333338277 +0000 UTC m=+1015.398579751" watchObservedRunningTime="2026-11-03 20:46:47.334611989 +0000 UTC m=+1015.399853462" Nov 03 20:46:48.234192 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:48.234154 5932 generic.go:356] "Generic (PLEG): container finished" podID="9ca762bf-f901-433d-9778-8f9ab063db33" containerID="425466fc2e646103fdf54872c4e31fd7bed335b6066f08740c07e6b0d0e97137" exitCode=0 Nov 03 20:46:48.234353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:48.234243 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" event={"ID":"9ca762bf-f901-433d-9778-8f9ab063db33","Type":"ContainerDied","Data":"425466fc2e646103fdf54872c4e31fd7bed335b6066f08740c07e6b0d0e97137"} Nov 03 20:46:49.389297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.389277 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:49.506790 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.506139 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca\") pod \"9ca762bf-f901-433d-9778-8f9ab063db33\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " Nov 03 20:46:49.506790 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.506199 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68\" (UniqueName: \"kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68\") pod \"9ca762bf-f901-433d-9778-8f9ab063db33\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " Nov 03 20:46:49.506790 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.506329 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca\") pod \"9ca762bf-f901-433d-9778-8f9ab063db33\" (UID: \"9ca762bf-f901-433d-9778-8f9ab063db33\") " Nov 03 20:46:49.507115 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.507089 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca" pod "9ca762bf-f901-433d-9778-8f9ab063db33" (UID: "9ca762bf-f901-433d-9778-8f9ab063db33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:46:49.508916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.508886 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca" pod "9ca762bf-f901-433d-9778-8f9ab063db33" (UID: "9ca762bf-f901-433d-9778-8f9ab063db33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:46:49.510049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.510019 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68" pod "9ca762bf-f901-433d-9778-8f9ab063db33" (UID: "9ca762bf-f901-433d-9778-8f9ab063db33"). InnerVolumeSpecName "kube-api-access-skl68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:46:49.607355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.607325 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:49.607355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.607350 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skl68\" (UniqueName: \"kubernetes.io/projected/9ca762bf-f901-433d-9778-8f9ab063db33-kube-api-access-skl68\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:49.607355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:49.607360 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ca762bf-f901-433d-9778-8f9ab063db33-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:46:50.248878 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:50.248851 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" Nov 03 20:46:50.249044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:50.248848 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx" event={"ID":"9ca762bf-f901-433d-9778-8f9ab063db33","Type":"ContainerDied","Data":"4189533107a75105728447dbabdd7762fbf4edde89990b33a0f89edd1abac9b8"} Nov 03 20:46:50.249044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:50.248982 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4189533107a75105728447dbabdd7762fbf4edde89990b33a0f89edd1abac9b8" Nov 03 20:46:56.787796 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.787762 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j"] Nov 03 20:46:56.788392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788374 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9ca762bf-f901-433d-9778-8f9ab063db33" containerName="revoke-keys" Nov 03 20:46:56.788467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788393 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca762bf-f901-433d-9778-8f9ab063db33" containerName="revoke-keys" Nov 03 20:46:56.788467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788424 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" containerName="revoke-keys" Nov 03 20:46:56.788467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788429 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" containerName="revoke-keys" Nov 03 20:46:56.788467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788443 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" containerName="revoke-keys" Nov 03 20:46:56.788467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788451 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" containerName="revoke-keys" Nov 03 20:46:56.788648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788558 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca762bf-f901-433d-9778-8f9ab063db33" containerName="revoke-keys" Nov 03 20:46:56.788648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788572 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" containerName="revoke-keys" Nov 03 20:46:56.788648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.788579 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" containerName="revoke-keys" Nov 03 20:46:56.792307 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.792290 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.841956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.841903 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:46:56.888098 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.888063 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:46:56.891322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.891297 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j"] Nov 03 20:46:56.891690 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.891664 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbzg\" (UniqueName: \"kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.891802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.891718 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.891874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.891858 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.993031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.992992 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.993226 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.993104 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbzg\" (UniqueName: \"kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.993226 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.993145 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.993853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.993820 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:56.995397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:56.995378 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:57.050563 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:57.050493 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbzg\" (UniqueName: \"kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg\") pod \"maas-api-revoke-sub-7e01f4d1-fcw8j\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:57.106015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:57.105985 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:46:57.485282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:57.485246 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j"] Nov 03 20:46:57.492052 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:57.492022 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4985f4_6814_463d_82f9_31b738108909.slice/crio-608811c2233e809c2a9087c678579482344c38b7bda8dad5a022e9c4feba4c30 WatchSource:0}: Error finding container 608811c2233e809c2a9087c678579482344c38b7bda8dad5a022e9c4feba4c30: Status 404 returned error can't find the container with id 608811c2233e809c2a9087c678579482344c38b7bda8dad5a022e9c4feba4c30 Nov 03 20:46:58.308614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.308570 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" event={"ID":"9c4985f4-6814-463d-82f9-31b738108909","Type":"ContainerStarted","Data":"8698ce80a33bdf0153abb2061c68c0f1da5408f5edcc287e70ba3b27a5280d7d"} Nov 03 20:46:58.309017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.308654 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" event={"ID":"9c4985f4-6814-463d-82f9-31b738108909","Type":"ContainerStarted","Data":"608811c2233e809c2a9087c678579482344c38b7bda8dad5a022e9c4feba4c30"} Nov 03 20:46:58.417378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.417319 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" podStartSLOduration=2.417300913 podStartE2EDuration="2.417300913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:46:58.4169744 +0000 UTC m=+1026.482215875" watchObservedRunningTime="2026-11-03 20:46:58.417300913 +0000 UTC m=+1026.482542388" Nov 03 20:46:58.725265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.725231 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw"] Nov 03 20:46:58.729429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.729410 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.783166 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.783135 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw"] Nov 03 20:46:58.814238 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.814207 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57t5\" (UniqueName: \"kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.814408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.814278 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.814458 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.814411 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.915537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.915498 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.915715 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.915686 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57t5\" (UniqueName: \"kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.915768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.915752 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.916212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.916193 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.918057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.918039 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:58.977165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:58.977095 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57t5\" (UniqueName: \"kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5\") pod \"maas-api-revoke-sub-4d39ed54-f9trw\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:59.042738 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.042708 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:46:59.307905 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.307815 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh"] Nov 03 20:46:59.318184 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.318156 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.324942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.324839 5932 generic.go:356] "Generic (PLEG): container finished" podID="9c4985f4-6814-463d-82f9-31b738108909" containerID="8698ce80a33bdf0153abb2061c68c0f1da5408f5edcc287e70ba3b27a5280d7d" exitCode=0 Nov 03 20:46:59.324942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.324876 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" event={"ID":"9c4985f4-6814-463d-82f9-31b738108909","Type":"ContainerDied","Data":"8698ce80a33bdf0153abb2061c68c0f1da5408f5edcc287e70ba3b27a5280d7d"} Nov 03 20:46:59.369036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.369003 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh"] Nov 03 20:46:59.421184 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.421143 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.421382 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.421286 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl9mn\" (UniqueName: \"kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.421477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.421456 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.479448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.479420 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw"] Nov 03 20:46:59.484957 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:46:59.484916 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ec0d12_02db_4f39_8cd8_fb84dae4cb94.slice/crio-fbb015795d619eff4b51f8d7477ed81cb8889298b3f1706a2c1b152ec207ad2a WatchSource:0}: Error finding container fbb015795d619eff4b51f8d7477ed81cb8889298b3f1706a2c1b152ec207ad2a: Status 404 returned error can't find the container with id fbb015795d619eff4b51f8d7477ed81cb8889298b3f1706a2c1b152ec207ad2a Nov 03 20:46:59.522776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.522745 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.522949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.522858 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.522949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.522927 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl9mn\" (UniqueName: \"kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.523392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.523372 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.525246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.525224 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.580377 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.580304 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl9mn\" (UniqueName: \"kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn\") pod \"maas-api-revoke-sub-9aef069b-99slh\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:46:59.635228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:46:59.635194 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:47:00.010095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.010050 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh"] Nov 03 20:47:00.015074 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:00.015046 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84dca63f_55ab_4e10_b46b_cd34fd1410ed.slice/crio-e317517308cc6bd8a3e70e4ea1c5f9dd3327d7d7a82a13ec1d0d928757a303bf WatchSource:0}: Error finding container e317517308cc6bd8a3e70e4ea1c5f9dd3327d7d7a82a13ec1d0d928757a303bf: Status 404 returned error can't find the container with id e317517308cc6bd8a3e70e4ea1c5f9dd3327d7d7a82a13ec1d0d928757a303bf Nov 03 20:47:00.333830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.333790 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" event={"ID":"84dca63f-55ab-4e10-b46b-cd34fd1410ed","Type":"ContainerStarted","Data":"e317517308cc6bd8a3e70e4ea1c5f9dd3327d7d7a82a13ec1d0d928757a303bf"} Nov 03 20:47:00.335737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.335707 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" event={"ID":"33ec0d12-02db-4f39-8cd8-fb84dae4cb94","Type":"ContainerStarted","Data":"b30934b0dc85470dd47493d7af7728ec020e30503d1174ff96f2656cdaaf2982"} Nov 03 20:47:00.335857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.335755 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" event={"ID":"33ec0d12-02db-4f39-8cd8-fb84dae4cb94","Type":"ContainerStarted","Data":"fbb015795d619eff4b51f8d7477ed81cb8889298b3f1706a2c1b152ec207ad2a"} Nov 03 20:47:00.446602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.446533 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" podStartSLOduration=2.446512899 podStartE2EDuration="2.446512899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:00.443117761 +0000 UTC m=+1028.508359236" watchObservedRunningTime="2026-11-03 20:47:00.446512899 +0000 UTC m=+1028.511754377" Nov 03 20:47:00.488534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.488476 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:47:00.639009 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.638967 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg\" (UniqueName: \"kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg\") pod \"9c4985f4-6814-463d-82f9-31b738108909\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " Nov 03 20:47:00.639129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.639110 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca\") pod \"9c4985f4-6814-463d-82f9-31b738108909\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " Nov 03 20:47:00.639252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.639236 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca\") pod \"9c4985f4-6814-463d-82f9-31b738108909\" (UID: \"9c4985f4-6814-463d-82f9-31b738108909\") " Nov 03 20:47:00.640038 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.640009 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca" pod "9c4985f4-6814-463d-82f9-31b738108909" (UID: "9c4985f4-6814-463d-82f9-31b738108909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:00.652255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.652218 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg" pod "9c4985f4-6814-463d-82f9-31b738108909" (UID: "9c4985f4-6814-463d-82f9-31b738108909"). InnerVolumeSpecName "kube-api-access-pcbzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:00.670676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.669600 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca" pod "9c4985f4-6814-463d-82f9-31b738108909" (UID: "9c4985f4-6814-463d-82f9-31b738108909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:00.740885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.740797 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4985f4-6814-463d-82f9-31b738108909-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:00.740885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.740824 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcbzg\" (UniqueName: \"kubernetes.io/projected/9c4985f4-6814-463d-82f9-31b738108909-kube-api-access-pcbzg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:00.740885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:00.740837 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4985f4-6814-463d-82f9-31b738108909-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:01.345964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.345879 5932 generic.go:356] "Generic (PLEG): container finished" podID="84dca63f-55ab-4e10-b46b-cd34fd1410ed" containerID="f8df41fe39f2756fd5dda64f245a9ca435e3616c8ee7c069e88e977e7fc537ea" exitCode=0 Nov 03 20:47:01.346357 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.345958 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" event={"ID":"84dca63f-55ab-4e10-b46b-cd34fd1410ed","Type":"ContainerDied","Data":"f8df41fe39f2756fd5dda64f245a9ca435e3616c8ee7c069e88e977e7fc537ea"} Nov 03 20:47:01.347729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.347702 5932 generic.go:356] "Generic (PLEG): container finished" podID="33ec0d12-02db-4f39-8cd8-fb84dae4cb94" containerID="b30934b0dc85470dd47493d7af7728ec020e30503d1174ff96f2656cdaaf2982" exitCode=0 Nov 03 20:47:01.347867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.347782 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" event={"ID":"33ec0d12-02db-4f39-8cd8-fb84dae4cb94","Type":"ContainerDied","Data":"b30934b0dc85470dd47493d7af7728ec020e30503d1174ff96f2656cdaaf2982"} Nov 03 20:47:01.349149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.349127 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" event={"ID":"9c4985f4-6814-463d-82f9-31b738108909","Type":"ContainerDied","Data":"608811c2233e809c2a9087c678579482344c38b7bda8dad5a022e9c4feba4c30"} Nov 03 20:47:01.349149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.349146 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j" Nov 03 20:47:01.349301 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:01.349162 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="608811c2233e809c2a9087c678579482344c38b7bda8dad5a022e9c4feba4c30" Nov 03 20:47:02.579992 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.579963 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:47:02.626792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.626768 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:47:02.764830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.764804 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn\" (UniqueName: \"kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn\") pod \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " Nov 03 20:47:02.764992 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.764884 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca\") pod \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " Nov 03 20:47:02.764992 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.764914 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca\") pod \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " Nov 03 20:47:02.764992 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.764959 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca\") pod \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " Nov 03 20:47:02.765140 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.765083 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5\" (UniqueName: \"kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5\") pod \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\" (UID: \"33ec0d12-02db-4f39-8cd8-fb84dae4cb94\") " Nov 03 20:47:02.765140 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.765126 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca\") pod \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\" (UID: \"84dca63f-55ab-4e10-b46b-cd34fd1410ed\") " Nov 03 20:47:02.765302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.765272 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca" pod "33ec0d12-02db-4f39-8cd8-fb84dae4cb94" (UID: "33ec0d12-02db-4f39-8cd8-fb84dae4cb94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:02.765431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.765335 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca" pod "84dca63f-55ab-4e10-b46b-cd34fd1410ed" (UID: "84dca63f-55ab-4e10-b46b-cd34fd1410ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:02.765697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.765675 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:02.765805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.765702 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:02.767168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.767141 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5" pod "33ec0d12-02db-4f39-8cd8-fb84dae4cb94" (UID: "33ec0d12-02db-4f39-8cd8-fb84dae4cb94"). InnerVolumeSpecName "kube-api-access-r57t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:02.767310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.767277 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca" pod "84dca63f-55ab-4e10-b46b-cd34fd1410ed" (UID: "84dca63f-55ab-4e10-b46b-cd34fd1410ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:02.767401 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.767344 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca" pod "33ec0d12-02db-4f39-8cd8-fb84dae4cb94" (UID: "33ec0d12-02db-4f39-8cd8-fb84dae4cb94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:02.767401 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.767350 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn" pod "84dca63f-55ab-4e10-b46b-cd34fd1410ed" (UID: "84dca63f-55ab-4e10-b46b-cd34fd1410ed"). InnerVolumeSpecName "kube-api-access-gl9mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:02.867247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.867202 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r57t5\" (UniqueName: \"kubernetes.io/projected/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-kube-api-access-r57t5\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:02.867247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.867245 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84dca63f-55ab-4e10-b46b-cd34fd1410ed-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:02.867247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.867256 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl9mn\" (UniqueName: \"kubernetes.io/projected/84dca63f-55ab-4e10-b46b-cd34fd1410ed-kube-api-access-gl9mn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:02.867481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:02.867266 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ec0d12-02db-4f39-8cd8-fb84dae4cb94-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:03.364428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:03.364398 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" Nov 03 20:47:03.364740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:03.364418 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh" event={"ID":"84dca63f-55ab-4e10-b46b-cd34fd1410ed","Type":"ContainerDied","Data":"e317517308cc6bd8a3e70e4ea1c5f9dd3327d7d7a82a13ec1d0d928757a303bf"} Nov 03 20:47:03.364870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:03.364773 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e317517308cc6bd8a3e70e4ea1c5f9dd3327d7d7a82a13ec1d0d928757a303bf" Nov 03 20:47:03.366135 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:03.366114 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" Nov 03 20:47:03.366247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:03.366109 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw" event={"ID":"33ec0d12-02db-4f39-8cd8-fb84dae4cb94","Type":"ContainerDied","Data":"fbb015795d619eff4b51f8d7477ed81cb8889298b3f1706a2c1b152ec207ad2a"} Nov 03 20:47:03.366299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:03.366267 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbb015795d619eff4b51f8d7477ed81cb8889298b3f1706a2c1b152ec207ad2a" Nov 03 20:47:05.783189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783150 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj"] Nov 03 20:47:05.783837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783820 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9c4985f4-6814-463d-82f9-31b738108909" containerName="revoke-keys" Nov 03 20:47:05.783837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783837 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4985f4-6814-463d-82f9-31b738108909" containerName="revoke-keys" Nov 03 20:47:05.783944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783860 5932 cpu_manager.go:414] "containerMap: removing container" podUID="33ec0d12-02db-4f39-8cd8-fb84dae4cb94" containerName="revoke-keys" Nov 03 20:47:05.783944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783865 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ec0d12-02db-4f39-8cd8-fb84dae4cb94" containerName="revoke-keys" Nov 03 20:47:05.783944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783873 5932 cpu_manager.go:414] "containerMap: removing container" podUID="84dca63f-55ab-4e10-b46b-cd34fd1410ed" containerName="revoke-keys" Nov 03 20:47:05.783944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783879 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84dca63f-55ab-4e10-b46b-cd34fd1410ed" containerName="revoke-keys" Nov 03 20:47:05.784059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783962 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4985f4-6814-463d-82f9-31b738108909" containerName="revoke-keys" Nov 03 20:47:05.784059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783975 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ec0d12-02db-4f39-8cd8-fb84dae4cb94" containerName="revoke-keys" Nov 03 20:47:05.784059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.783982 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="84dca63f-55ab-4e10-b46b-cd34fd1410ed" containerName="revoke-keys" Nov 03 20:47:05.787798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.787779 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.796586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.796561 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.796696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.796659 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm5qm\" (UniqueName: \"kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.796757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.796704 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.882538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.882513 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:05.882538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.882509 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:47:05.886429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.886406 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj"] Nov 03 20:47:05.902119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.902083 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.902243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.902162 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm5qm\" (UniqueName: \"kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.902243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.902205 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.903462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.903055 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.908205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.908177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:05.971646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:05.971603 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm5qm\" (UniqueName: \"kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm\") pod \"maas-api-revoke-sub-9b12aa87-sgwdj\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:06.102853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:06.102751 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:06.559215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:06.558748 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj"] Nov 03 20:47:07.407394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:07.406223 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" event={"ID":"149d7d57-4fdb-47e2-a29d-33a7c10e848f","Type":"ContainerStarted","Data":"578a3c9294144945ceeb56f67fcd400d1927a692916f065c3428268f0b90720b"} Nov 03 20:47:07.407394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:07.406289 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" event={"ID":"149d7d57-4fdb-47e2-a29d-33a7c10e848f","Type":"ContainerStarted","Data":"81245a95d84882d3d19361c194722fb32b590b931fe350992d3fe6bcb3a8fff8"} Nov 03 20:47:07.549647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:07.547339 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" podStartSLOduration=2.547321964 podStartE2EDuration="2.547321964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:07.520534215 +0000 UTC m=+1035.585775689" watchObservedRunningTime="2026-11-03 20:47:07.547321964 +0000 UTC m=+1035.612563437" Nov 03 20:47:08.416983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:08.416949 5932 generic.go:356] "Generic (PLEG): container finished" podID="149d7d57-4fdb-47e2-a29d-33a7c10e848f" containerID="578a3c9294144945ceeb56f67fcd400d1927a692916f065c3428268f0b90720b" exitCode=0 Nov 03 20:47:08.417349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:08.417035 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" event={"ID":"149d7d57-4fdb-47e2-a29d-33a7c10e848f","Type":"ContainerDied","Data":"578a3c9294144945ceeb56f67fcd400d1927a692916f065c3428268f0b90720b"} Nov 03 20:47:09.578025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.578001 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:09.682043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.681985 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca\") pod \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " Nov 03 20:47:09.682168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.682073 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm\" (UniqueName: \"kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm\") pod \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " Nov 03 20:47:09.682229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.682173 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca\") pod \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\" (UID: \"149d7d57-4fdb-47e2-a29d-33a7c10e848f\") " Nov 03 20:47:09.682363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.682335 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca" pod "149d7d57-4fdb-47e2-a29d-33a7c10e848f" (UID: "149d7d57-4fdb-47e2-a29d-33a7c10e848f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:09.682668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.682508 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.684116 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.684094 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm" pod "149d7d57-4fdb-47e2-a29d-33a7c10e848f" (UID: "149d7d57-4fdb-47e2-a29d-33a7c10e848f"). InnerVolumeSpecName "kube-api-access-hm5qm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:09.684206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.684139 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca" pod "149d7d57-4fdb-47e2-a29d-33a7c10e848f" (UID: "149d7d57-4fdb-47e2-a29d-33a7c10e848f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:09.783842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.783810 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149d7d57-4fdb-47e2-a29d-33a7c10e848f-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:09.783842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:09.783837 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm5qm\" (UniqueName: \"kubernetes.io/projected/149d7d57-4fdb-47e2-a29d-33a7c10e848f-kube-api-access-hm5qm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:10.434455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:10.434399 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" event={"ID":"149d7d57-4fdb-47e2-a29d-33a7c10e848f","Type":"ContainerDied","Data":"81245a95d84882d3d19361c194722fb32b590b931fe350992d3fe6bcb3a8fff8"} Nov 03 20:47:10.434689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:10.434510 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81245a95d84882d3d19361c194722fb32b590b931fe350992d3fe6bcb3a8fff8" Nov 03 20:47:10.434689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:10.434604 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj" Nov 03 20:47:13.246214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.246183 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5"] Nov 03 20:47:13.246877 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.246860 5932 cpu_manager.go:414] "containerMap: removing container" podUID="149d7d57-4fdb-47e2-a29d-33a7c10e848f" containerName="revoke-keys" Nov 03 20:47:13.246926 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.246878 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149d7d57-4fdb-47e2-a29d-33a7c10e848f" containerName="revoke-keys" Nov 03 20:47:13.246984 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.246975 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="149d7d57-4fdb-47e2-a29d-33a7c10e848f" containerName="revoke-keys" Nov 03 20:47:13.250423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.250408 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.299675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.299636 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:13.304280 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.304249 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5"] Nov 03 20:47:13.322872 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.322840 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx4hc\" (UniqueName: \"kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.323084 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.322991 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.323084 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.323029 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.345757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.345725 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:47:13.423901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.423869 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.423901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.423905 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.424117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.424096 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx4hc\" (UniqueName: \"kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.424652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.424602 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.426386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.426367 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.481218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.481187 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx4hc\" (UniqueName: \"kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc\") pod \"maas-api-revoke-sub-33d5d111-5cds5\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.579663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.579585 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:13.953573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:13.953536 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5"] Nov 03 20:47:13.959218 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:13.959187 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod594275a5_5aa9_4cd8_8f96_dd5ff5e7a197.slice/crio-1340c238e1806e388e5647de27441163cb441598644ae9ce986538192e377d04 WatchSource:0}: Error finding container 1340c238e1806e388e5647de27441163cb441598644ae9ce986538192e377d04: Status 404 returned error can't find the container with id 1340c238e1806e388e5647de27441163cb441598644ae9ce986538192e377d04 Nov 03 20:47:14.158714 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:47:14.158669 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:47:14.467327 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:14.467286 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" event={"ID":"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197","Type":"ContainerStarted","Data":"d1e6eb81fccb61319a65f65acd465dd32806b174a02e9e399f9cd06469b522da"} Nov 03 20:47:14.467769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:14.467392 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" event={"ID":"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197","Type":"ContainerStarted","Data":"1340c238e1806e388e5647de27441163cb441598644ae9ce986538192e377d04"} Nov 03 20:47:14.575992 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:14.575868 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" podStartSLOduration=1.5758501059999999 podStartE2EDuration="1.575850106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:14.574855562 +0000 UTC m=+1042.640097034" watchObservedRunningTime="2026-11-03 20:47:14.575850106 +0000 UTC m=+1042.641091582" Nov 03 20:47:15.214635 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.214588 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w"] Nov 03 20:47:15.219006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.218981 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.242618 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.242598 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.242732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.242692 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.242870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.242849 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9vf\" (UniqueName: \"kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.298855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.298824 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w"] Nov 03 20:47:15.343868 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.343842 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.344040 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.343950 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.344126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.344099 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9vf\" (UniqueName: \"kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.344749 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.344729 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.346219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.346200 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.401216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.401192 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9vf\" (UniqueName: \"kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf\") pod \"maas-api-revoke-sub-3fe72be6-7hr6w\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.476004 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.475924 5932 generic.go:356] "Generic (PLEG): container finished" podID="594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" containerID="d1e6eb81fccb61319a65f65acd465dd32806b174a02e9e399f9cd06469b522da" exitCode=0 Nov 03 20:47:15.476351 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.476011 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" event={"ID":"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197","Type":"ContainerDied","Data":"d1e6eb81fccb61319a65f65acd465dd32806b174a02e9e399f9cd06469b522da"} Nov 03 20:47:15.532063 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.532039 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:15.904689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:15.904656 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w"] Nov 03 20:47:15.910350 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:15.910325 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff087102_1bb8_41ce_ae9e_09d2477d8560.slice/crio-96a0516b7c1e5254e24fe2fdb1fdb5f3e46d7f985ea6d6282441b41c85454dee WatchSource:0}: Error finding container 96a0516b7c1e5254e24fe2fdb1fdb5f3e46d7f985ea6d6282441b41c85454dee: Status 404 returned error can't find the container with id 96a0516b7c1e5254e24fe2fdb1fdb5f3e46d7f985ea6d6282441b41c85454dee Nov 03 20:47:16.475183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.475151 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj"] Nov 03 20:47:16.479446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.479427 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.485369 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.485307 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" event={"ID":"ff087102-1bb8-41ce-ae9e-09d2477d8560","Type":"ContainerStarted","Data":"3a218c2227e2cce9ca837dde6c4407ba2a9be7da1756e69232280938a15f0487"} Nov 03 20:47:16.485369 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.485352 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" event={"ID":"ff087102-1bb8-41ce-ae9e-09d2477d8560","Type":"ContainerStarted","Data":"96a0516b7c1e5254e24fe2fdb1fdb5f3e46d7f985ea6d6282441b41c85454dee"} Nov 03 20:47:16.532902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.532869 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj"] Nov 03 20:47:16.556322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.556294 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.556464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.556396 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4mn\" (UniqueName: \"kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.556464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.556450 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.652879 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.652856 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:16.658120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.658098 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4mn\" (UniqueName: \"kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.658205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.658144 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.658269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.658255 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.658805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.658784 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.660405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.660384 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.696967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.696917 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" podStartSLOduration=1.6969054799999999 podStartE2EDuration="1.69690548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:16.694374351 +0000 UTC m=+1044.759615823" watchObservedRunningTime="2026-11-03 20:47:16.69690548 +0000 UTC m=+1044.762146952" Nov 03 20:47:16.715282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.715249 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4mn\" (UniqueName: \"kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn\") pod \"maas-api-revoke-sub-308ecf28-fxnwj\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.758920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.758855 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca\") pod \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " Nov 03 20:47:16.759030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.758994 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca\") pod \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " Nov 03 20:47:16.759030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.759020 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc\" (UniqueName: \"kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc\") pod \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\" (UID: \"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197\") " Nov 03 20:47:16.759211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.759186 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca" pod "594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" (UID: "594275a5-5aa9-4cd8-8f96-dd5ff5e7a197"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:16.759365 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.759346 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:16.761007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.760984 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca" pod "594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" (UID: "594275a5-5aa9-4cd8-8f96-dd5ff5e7a197"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:16.761204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.761187 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc" pod "594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" (UID: "594275a5-5aa9-4cd8-8f96-dd5ff5e7a197"). InnerVolumeSpecName "kube-api-access-dx4hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:16.795190 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.795171 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:16.860893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.860863 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:16.861022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:16.860898 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx4hc\" (UniqueName: \"kubernetes.io/projected/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197-kube-api-access-dx4hc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:17.172997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.172964 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj"] Nov 03 20:47:17.177610 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:17.177584 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379fcd8e_1f75_45d2_a5e5_5ba6e562d068.slice/crio-27bb3ead6764d1606abbb09f7d3f5666b6b8f0e5582b96f1391e01353ce8d4ff WatchSource:0}: Error finding container 27bb3ead6764d1606abbb09f7d3f5666b6b8f0e5582b96f1391e01353ce8d4ff: Status 404 returned error can't find the container with id 27bb3ead6764d1606abbb09f7d3f5666b6b8f0e5582b96f1391e01353ce8d4ff Nov 03 20:47:17.501801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.501771 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" Nov 03 20:47:17.501801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.501793 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5" event={"ID":"594275a5-5aa9-4cd8-8f96-dd5ff5e7a197","Type":"ContainerDied","Data":"1340c238e1806e388e5647de27441163cb441598644ae9ce986538192e377d04"} Nov 03 20:47:17.502236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.501860 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1340c238e1806e388e5647de27441163cb441598644ae9ce986538192e377d04" Nov 03 20:47:17.503699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.503669 5932 generic.go:356] "Generic (PLEG): container finished" podID="ff087102-1bb8-41ce-ae9e-09d2477d8560" containerID="3a218c2227e2cce9ca837dde6c4407ba2a9be7da1756e69232280938a15f0487" exitCode=0 Nov 03 20:47:17.503854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.503701 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" event={"ID":"ff087102-1bb8-41ce-ae9e-09d2477d8560","Type":"ContainerDied","Data":"3a218c2227e2cce9ca837dde6c4407ba2a9be7da1756e69232280938a15f0487"} Nov 03 20:47:17.505408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.505384 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" event={"ID":"379fcd8e-1f75-45d2-a5e5-5ba6e562d068","Type":"ContainerStarted","Data":"777ca4f807ebd18ba9b41c10c602b306bfb348b5920f0efd8d00b2ca260e4b92"} Nov 03 20:47:17.505544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.505421 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" event={"ID":"379fcd8e-1f75-45d2-a5e5-5ba6e562d068","Type":"ContainerStarted","Data":"27bb3ead6764d1606abbb09f7d3f5666b6b8f0e5582b96f1391e01353ce8d4ff"} Nov 03 20:47:17.789471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:17.788235 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" podStartSLOduration=1.788214615 podStartE2EDuration="1.788214615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:17.784826951 +0000 UTC m=+1045.850068425" watchObservedRunningTime="2026-11-03 20:47:17.788214615 +0000 UTC m=+1045.853456089" Nov 03 20:47:18.513826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.513793 5932 generic.go:356] "Generic (PLEG): container finished" podID="379fcd8e-1f75-45d2-a5e5-5ba6e562d068" containerID="777ca4f807ebd18ba9b41c10c602b306bfb348b5920f0efd8d00b2ca260e4b92" exitCode=0 Nov 03 20:47:18.514199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.513881 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" event={"ID":"379fcd8e-1f75-45d2-a5e5-5ba6e562d068","Type":"ContainerDied","Data":"777ca4f807ebd18ba9b41c10c602b306bfb348b5920f0efd8d00b2ca260e4b92"} Nov 03 20:47:18.658849 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.658822 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:18.697122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.697095 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca\") pod \"ff087102-1bb8-41ce-ae9e-09d2477d8560\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " Nov 03 20:47:18.697259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.697197 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca\") pod \"ff087102-1bb8-41ce-ae9e-09d2477d8560\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " Nov 03 20:47:18.697319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.697295 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf\" (UniqueName: \"kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf\") pod \"ff087102-1bb8-41ce-ae9e-09d2477d8560\" (UID: \"ff087102-1bb8-41ce-ae9e-09d2477d8560\") " Nov 03 20:47:18.697675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.697596 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca" pod "ff087102-1bb8-41ce-ae9e-09d2477d8560" (UID: "ff087102-1bb8-41ce-ae9e-09d2477d8560"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:18.697798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.697778 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:18.699209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.699181 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca" pod "ff087102-1bb8-41ce-ae9e-09d2477d8560" (UID: "ff087102-1bb8-41ce-ae9e-09d2477d8560"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:18.699399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.699378 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf" pod "ff087102-1bb8-41ce-ae9e-09d2477d8560" (UID: "ff087102-1bb8-41ce-ae9e-09d2477d8560"). InnerVolumeSpecName "kube-api-access-rf9vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:18.798959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.798872 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9vf\" (UniqueName: \"kubernetes.io/projected/ff087102-1bb8-41ce-ae9e-09d2477d8560-kube-api-access-rf9vf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:18.798959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:18.798911 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff087102-1bb8-41ce-ae9e-09d2477d8560-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:19.526136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.526114 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" Nov 03 20:47:19.526470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.526148 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w" event={"ID":"ff087102-1bb8-41ce-ae9e-09d2477d8560","Type":"ContainerDied","Data":"96a0516b7c1e5254e24fe2fdb1fdb5f3e46d7f985ea6d6282441b41c85454dee"} Nov 03 20:47:19.526470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.526197 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96a0516b7c1e5254e24fe2fdb1fdb5f3e46d7f985ea6d6282441b41c85454dee" Nov 03 20:47:19.683952 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.683926 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:19.809168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.809086 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca\") pod \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " Nov 03 20:47:19.809168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.809134 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn\" (UniqueName: \"kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn\") pod \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " Nov 03 20:47:19.809413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.809185 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca\") pod \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\" (UID: \"379fcd8e-1f75-45d2-a5e5-5ba6e562d068\") " Nov 03 20:47:19.809413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.809375 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca" pod "379fcd8e-1f75-45d2-a5e5-5ba6e562d068" (UID: "379fcd8e-1f75-45d2-a5e5-5ba6e562d068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:19.809694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.809677 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:19.811319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.811298 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca" pod "379fcd8e-1f75-45d2-a5e5-5ba6e562d068" (UID: "379fcd8e-1f75-45d2-a5e5-5ba6e562d068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:19.811319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.811309 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn" pod "379fcd8e-1f75-45d2-a5e5-5ba6e562d068" (UID: "379fcd8e-1f75-45d2-a5e5-5ba6e562d068"). InnerVolumeSpecName "kube-api-access-wn4mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:19.910426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.910390 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn4mn\" (UniqueName: \"kubernetes.io/projected/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-kube-api-access-wn4mn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:19.910426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:19.910419 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379fcd8e-1f75-45d2-a5e5-5ba6e562d068-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:20.541860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:20.541826 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" Nov 03 20:47:20.542255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:20.541853 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj" event={"ID":"379fcd8e-1f75-45d2-a5e5-5ba6e562d068","Type":"ContainerDied","Data":"27bb3ead6764d1606abbb09f7d3f5666b6b8f0e5582b96f1391e01353ce8d4ff"} Nov 03 20:47:20.542255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:20.541930 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27bb3ead6764d1606abbb09f7d3f5666b6b8f0e5582b96f1391e01353ce8d4ff" Nov 03 20:47:24.002146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002110 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn"] Nov 03 20:47:24.002747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002717 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ff087102-1bb8-41ce-ae9e-09d2477d8560" containerName="revoke-keys" Nov 03 20:47:24.002803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002748 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff087102-1bb8-41ce-ae9e-09d2477d8560" containerName="revoke-keys" Nov 03 20:47:24.002803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002762 5932 cpu_manager.go:414] "containerMap: removing container" podUID="379fcd8e-1f75-45d2-a5e5-5ba6e562d068" containerName="revoke-keys" Nov 03 20:47:24.002803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002767 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379fcd8e-1f75-45d2-a5e5-5ba6e562d068" containerName="revoke-keys" Nov 03 20:47:24.002803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002780 5932 cpu_manager.go:414] "containerMap: removing container" podUID="594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" containerName="revoke-keys" Nov 03 20:47:24.002803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002788 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" containerName="revoke-keys" Nov 03 20:47:24.002950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002875 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" containerName="revoke-keys" Nov 03 20:47:24.002950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002891 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff087102-1bb8-41ce-ae9e-09d2477d8560" containerName="revoke-keys" Nov 03 20:47:24.002950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.002899 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="379fcd8e-1f75-45d2-a5e5-5ba6e562d068" containerName="revoke-keys" Nov 03 20:47:24.006551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.006535 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.052033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.052007 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.052161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.052049 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hlcd\" (UniqueName: \"kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.052205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.052196 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.056063 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.056042 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:24.060825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.060802 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn"] Nov 03 20:47:24.102353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.102329 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:47:24.153465 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.153434 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.153602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.153579 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.153687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.153647 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hlcd\" (UniqueName: \"kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.154208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.154182 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.155823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.155805 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.210856 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.210828 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hlcd\" (UniqueName: \"kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd\") pod \"maas-api-revoke-sub-dc27b964-w4mhn\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.320786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.320716 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:24.705149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:24.705113 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn"] Nov 03 20:47:24.712415 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:24.712388 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc2914e_7eb4_4375_a223_83854e55fd08.slice/crio-25d15f5d886706774143ed705c7924e5d3ae1976555698669ea255df446309e8 WatchSource:0}: Error finding container 25d15f5d886706774143ed705c7924e5d3ae1976555698669ea255df446309e8: Status 404 returned error can't find the container with id 25d15f5d886706774143ed705c7924e5d3ae1976555698669ea255df446309e8 Nov 03 20:47:25.588784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:25.588744 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" event={"ID":"adc2914e-7eb4-4375-a223-83854e55fd08","Type":"ContainerStarted","Data":"ee373e99270323853147fb1fcb32dc278abc40ecd1bc3f13ac3c2767df470af3"} Nov 03 20:47:25.589304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:25.588848 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" event={"ID":"adc2914e-7eb4-4375-a223-83854e55fd08","Type":"ContainerStarted","Data":"25d15f5d886706774143ed705c7924e5d3ae1976555698669ea255df446309e8"} Nov 03 20:47:25.704406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:25.704338 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" podStartSLOduration=2.704318773 podStartE2EDuration="2.704318773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:25.703882054 +0000 UTC m=+1053.769123529" watchObservedRunningTime="2026-11-03 20:47:25.704318773 +0000 UTC m=+1053.769560247" Nov 03 20:47:26.597184 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:26.597147 5932 generic.go:356] "Generic (PLEG): container finished" podID="adc2914e-7eb4-4375-a223-83854e55fd08" containerID="ee373e99270323853147fb1fcb32dc278abc40ecd1bc3f13ac3c2767df470af3" exitCode=0 Nov 03 20:47:26.597539 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:26.597205 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" event={"ID":"adc2914e-7eb4-4375-a223-83854e55fd08","Type":"ContainerDied","Data":"ee373e99270323853147fb1fcb32dc278abc40ecd1bc3f13ac3c2767df470af3"} Nov 03 20:47:27.751419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.751393 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:27.930360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.930328 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca\") pod \"adc2914e-7eb4-4375-a223-83854e55fd08\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " Nov 03 20:47:27.930534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.930406 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd\" (UniqueName: \"kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd\") pod \"adc2914e-7eb4-4375-a223-83854e55fd08\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " Nov 03 20:47:27.930639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.930558 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca\") pod \"adc2914e-7eb4-4375-a223-83854e55fd08\" (UID: \"adc2914e-7eb4-4375-a223-83854e55fd08\") " Nov 03 20:47:27.930752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.930724 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca" pod "adc2914e-7eb4-4375-a223-83854e55fd08" (UID: "adc2914e-7eb4-4375-a223-83854e55fd08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:27.931006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.930983 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:27.932659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.932610 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca" pod "adc2914e-7eb4-4375-a223-83854e55fd08" (UID: "adc2914e-7eb4-4375-a223-83854e55fd08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:27.932778 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:27.932683 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd" pod "adc2914e-7eb4-4375-a223-83854e55fd08" (UID: "adc2914e-7eb4-4375-a223-83854e55fd08"). InnerVolumeSpecName "kube-api-access-7hlcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:28.032228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:28.032193 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc2914e-7eb4-4375-a223-83854e55fd08-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:28.032228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:28.032224 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hlcd\" (UniqueName: \"kubernetes.io/projected/adc2914e-7eb4-4375-a223-83854e55fd08-kube-api-access-7hlcd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:28.611772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:28.611743 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" Nov 03 20:47:28.611941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:28.611780 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn" event={"ID":"adc2914e-7eb4-4375-a223-83854e55fd08","Type":"ContainerDied","Data":"25d15f5d886706774143ed705c7924e5d3ae1976555698669ea255df446309e8"} Nov 03 20:47:28.611941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:28.611828 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d15f5d886706774143ed705c7924e5d3ae1976555698669ea255df446309e8" Nov 03 20:47:32.052954 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.052922 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s"] Nov 03 20:47:32.053471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.053455 5932 cpu_manager.go:414] "containerMap: removing container" podUID="adc2914e-7eb4-4375-a223-83854e55fd08" containerName="revoke-keys" Nov 03 20:47:32.053544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.053471 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc2914e-7eb4-4375-a223-83854e55fd08" containerName="revoke-keys" Nov 03 20:47:32.053600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.053560 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc2914e-7eb4-4375-a223-83854e55fd08" containerName="revoke-keys" Nov 03 20:47:32.058153 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.058135 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.078059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.078031 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjwmq\" (UniqueName: \"kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.078199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.078137 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.078264 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.078224 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.107431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.107334 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s"] Nov 03 20:47:32.148103 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.148073 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:32.148273 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.148201 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:47:32.179866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.179831 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjwmq\" (UniqueName: \"kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.180043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.179880 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.180043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.179918 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.180588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.180559 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.182255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.182232 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.237088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.237054 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjwmq\" (UniqueName: \"kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq\") pod \"maas-api-revoke-sub-28b3ba55-9ct5s\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.372707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.372611 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:32.767769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:32.767739 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s"] Nov 03 20:47:32.771785 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:32.771757 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a837079_380b_4a95_b7a8_3bfbb15d5609.slice/crio-cf10c5cab40b1421f10320232c81a56e73d42af898af6f3aef3336af66e2b118 WatchSource:0}: Error finding container cf10c5cab40b1421f10320232c81a56e73d42af898af6f3aef3336af66e2b118: Status 404 returned error can't find the container with id cf10c5cab40b1421f10320232c81a56e73d42af898af6f3aef3336af66e2b118 Nov 03 20:47:33.650463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:33.650429 5932 generic.go:356] "Generic (PLEG): container finished" podID="2a837079-380b-4a95-b7a8-3bfbb15d5609" containerID="6c9867f045d30f96ddb2f267873ebcf9c3a5d341e048bf02c037463b912ca75d" exitCode=0 Nov 03 20:47:33.650819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:33.650515 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" event={"ID":"2a837079-380b-4a95-b7a8-3bfbb15d5609","Type":"ContainerDied","Data":"6c9867f045d30f96ddb2f267873ebcf9c3a5d341e048bf02c037463b912ca75d"} Nov 03 20:47:33.650819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:33.650565 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" event={"ID":"2a837079-380b-4a95-b7a8-3bfbb15d5609","Type":"ContainerStarted","Data":"cf10c5cab40b1421f10320232c81a56e73d42af898af6f3aef3336af66e2b118"} Nov 03 20:47:34.472646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.472603 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-ggs45-tf5xz"] Nov 03 20:47:34.476215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.476196 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.505842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.505815 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.505931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.505869 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kjlk\" (UniqueName: \"kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.506039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.506019 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.506083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.506051 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.580988 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.580962 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ggs45-tf5xz"] Nov 03 20:47:34.606755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.606728 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.606857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.606774 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kjlk\" (UniqueName: \"kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.606924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.606907 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.606967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.606939 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.607323 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.607301 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.607663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.607643 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.607663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.607655 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.664473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.664452 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kjlk\" (UniqueName: \"kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk\") pod \"periodic-gathering-ggs45-tf5xz\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.790271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.790190 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:47:34.840039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.840008 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:34.910249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.910226 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq\" (UniqueName: \"kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq\") pod \"2a837079-380b-4a95-b7a8-3bfbb15d5609\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " Nov 03 20:47:34.910384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.910255 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca\") pod \"2a837079-380b-4a95-b7a8-3bfbb15d5609\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " Nov 03 20:47:34.910384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.910346 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca\") pod \"2a837079-380b-4a95-b7a8-3bfbb15d5609\" (UID: \"2a837079-380b-4a95-b7a8-3bfbb15d5609\") " Nov 03 20:47:34.910685 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.910655 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca" pod "2a837079-380b-4a95-b7a8-3bfbb15d5609" (UID: "2a837079-380b-4a95-b7a8-3bfbb15d5609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:34.912176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.912155 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq" pod "2a837079-380b-4a95-b7a8-3bfbb15d5609" (UID: "2a837079-380b-4a95-b7a8-3bfbb15d5609"). InnerVolumeSpecName "kube-api-access-qjwmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:34.912253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:34.912239 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca" pod "2a837079-380b-4a95-b7a8-3bfbb15d5609" (UID: "2a837079-380b-4a95-b7a8-3bfbb15d5609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:35.012154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.012121 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjwmq\" (UniqueName: \"kubernetes.io/projected/2a837079-380b-4a95-b7a8-3bfbb15d5609-kube-api-access-qjwmq\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:35.012154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.012153 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:35.012341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.012164 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a837079-380b-4a95-b7a8-3bfbb15d5609-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:35.172599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.172561 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-ggs45-tf5xz"] Nov 03 20:47:35.177073 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:35.177045 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae58f20c_eff4_47e6_b351_4a8bc416849b.slice/crio-1a94ef18cf05fbb46b4d7101df7f8dfb33fde3cd6c2acd80712367f9a6ba9d0f WatchSource:0}: Error finding container 1a94ef18cf05fbb46b4d7101df7f8dfb33fde3cd6c2acd80712367f9a6ba9d0f: Status 404 returned error can't find the container with id 1a94ef18cf05fbb46b4d7101df7f8dfb33fde3cd6c2acd80712367f9a6ba9d0f Nov 03 20:47:35.667332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.667304 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" Nov 03 20:47:35.667806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.667340 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s" event={"ID":"2a837079-380b-4a95-b7a8-3bfbb15d5609","Type":"ContainerDied","Data":"cf10c5cab40b1421f10320232c81a56e73d42af898af6f3aef3336af66e2b118"} Nov 03 20:47:35.667806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.667396 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf10c5cab40b1421f10320232c81a56e73d42af898af6f3aef3336af66e2b118" Nov 03 20:47:35.669106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.669080 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" event={"ID":"ae58f20c-eff4-47e6-b351-4a8bc416849b","Type":"ContainerStarted","Data":"63185253cfdfca982d43dd8738b9122d7172dd3e5760fb93760ac7b3c1ffcf5b"} Nov 03 20:47:35.669230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.669128 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" event={"ID":"ae58f20c-eff4-47e6-b351-4a8bc416849b","Type":"ContainerStarted","Data":"1a94ef18cf05fbb46b4d7101df7f8dfb33fde3cd6c2acd80712367f9a6ba9d0f"} Nov 03 20:47:35.780087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:35.780042 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" podStartSLOduration=1.780029239 podStartE2EDuration="1.780029239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:35.776800129 +0000 UTC m=+1063.842041605" watchObservedRunningTime="2026-11-03 20:47:35.780029239 +0000 UTC m=+1063.845270711" Nov 03 20:47:38.890214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.890168 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm"] Nov 03 20:47:38.891016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.890999 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2a837079-380b-4a95-b7a8-3bfbb15d5609" containerName="revoke-keys" Nov 03 20:47:38.891075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.891018 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a837079-380b-4a95-b7a8-3bfbb15d5609" containerName="revoke-keys" Nov 03 20:47:38.891123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.891114 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a837079-380b-4a95-b7a8-3bfbb15d5609" containerName="revoke-keys" Nov 03 20:47:38.894551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.894531 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.957997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.957934 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958133 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958080 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958137 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958220 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958277 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958413 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958496 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958458 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnkr\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958561 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958507 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.958701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.958609 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:38.995584 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:38.995552 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm"] Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061065 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061127 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061173 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061212 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061305 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061350 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnkr\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061399 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061435 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.061647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.061491 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.062238 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.062054 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.068656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.062317 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.068656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.063529 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.068656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.063828 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.068656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.066207 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.068656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.067406 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.083651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.082151 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.127392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.126287 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.135574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.135541 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnkr\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.211117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.211035 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:39.644999 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.644971 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm"] Nov 03 20:47:39.648822 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:39.648797 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376ffb76_c668_4150_b049_14ef5689f951.slice/crio-c454801a54a07b27264bcc4b4bad200564d31bd7010d00548c785d2bed57ba75 WatchSource:0}: Error finding container c454801a54a07b27264bcc4b4bad200564d31bd7010d00548c785d2bed57ba75: Status 404 returned error can't find the container with id c454801a54a07b27264bcc4b4bad200564d31bd7010d00548c785d2bed57ba75 Nov 03 20:47:39.651183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.651151 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:47:39.651255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.651238 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:47:39.651301 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.651285 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:47:39.722890 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.722861 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" event={"ID":"376ffb76-c668-4150-b049-14ef5689f951","Type":"ContainerStarted","Data":"de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b"} Nov 03 20:47:39.723026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.722916 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" event={"ID":"376ffb76-c668-4150-b049-14ef5689f951","Type":"ContainerStarted","Data":"c454801a54a07b27264bcc4b4bad200564d31bd7010d00548c785d2bed57ba75"} Nov 03 20:47:39.845261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:39.845209 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podStartSLOduration=1.8451948740000002 podStartE2EDuration="1.845194874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:39.841278658 +0000 UTC m=+1067.906520132" watchObservedRunningTime="2026-11-03 20:47:39.845194874 +0000 UTC m=+1067.910436347" Nov 03 20:47:40.211707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:40.211668 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:40.213331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:40.213308 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:40.213427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:40.213361 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:41.212021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:41.211985 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:41.212402 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:41.212038 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:42.211999 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:42.211964 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:42.212152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:42.212018 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:43.211433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:43.211394 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:43.211605 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:43.211455 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:44.172062 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:47:44.172023 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:47:44.211661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:44.211613 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:44.211830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:44.211680 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:45.211526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.211492 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:45.212003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.211558 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:45.375382 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.375346 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g"] Nov 03 20:47:45.379853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.379835 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.451268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.450903 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g"] Nov 03 20:47:45.457524 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.457481 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pszzr\" (UniqueName: \"kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.457742 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.457578 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.457833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.457746 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.487188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.487091 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:47:45.487188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.487142 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:47:45.559086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.559055 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pszzr\" (UniqueName: \"kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.559284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.559110 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.559284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.559187 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.560180 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.560150 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.561544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.561524 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.616718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.616691 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pszzr\" (UniqueName: \"kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr\") pod \"maas-api-revoke-sub-57ccf053-c8n4g\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:45.694638 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:45.694590 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:46.075410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:46.075323 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g"] Nov 03 20:47:46.080274 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:46.080234 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcc03fd_9fa9_41f4_a46c_48e637963102.slice/crio-af7bcb104ec3d2c29fd0e9ec2e35279d323a59c67fdc37ec8ce5002c23884442 WatchSource:0}: Error finding container af7bcb104ec3d2c29fd0e9ec2e35279d323a59c67fdc37ec8ce5002c23884442: Status 404 returned error can't find the container with id af7bcb104ec3d2c29fd0e9ec2e35279d323a59c67fdc37ec8ce5002c23884442 Nov 03 20:47:46.211720 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:46.211659 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Nov 03 20:47:46.211720 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:46.211715 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Nov 03 20:47:46.797763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:46.797381 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" event={"ID":"cbcc03fd-9fa9-41f4-a46c-48e637963102","Type":"ContainerStarted","Data":"566fff1e82780550e6b59fec0741246f231483c885c5806548a1d245c1bad207"} Nov 03 20:47:46.797763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:46.797449 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" event={"ID":"cbcc03fd-9fa9-41f4-a46c-48e637963102","Type":"ContainerStarted","Data":"af7bcb104ec3d2c29fd0e9ec2e35279d323a59c67fdc37ec8ce5002c23884442"} Nov 03 20:47:46.908536 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:46.908487 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" podStartSLOduration=1.908471578 podStartE2EDuration="1.908471578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:46.905926279 +0000 UTC m=+1074.971167756" watchObservedRunningTime="2026-11-03 20:47:46.908471578 +0000 UTC m=+1074.973713056" Nov 03 20:47:47.216809 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:47.216780 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:47.217243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:47.217175 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:47.217768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:47.217753 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:47:47.805921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:47.805890 5932 generic.go:356] "Generic (PLEG): container finished" podID="cbcc03fd-9fa9-41f4-a46c-48e637963102" containerID="566fff1e82780550e6b59fec0741246f231483c885c5806548a1d245c1bad207" exitCode=0 Nov 03 20:47:47.806092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:47.805971 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" event={"ID":"cbcc03fd-9fa9-41f4-a46c-48e637963102","Type":"ContainerDied","Data":"566fff1e82780550e6b59fec0741246f231483c885c5806548a1d245c1bad207"} Nov 03 20:47:48.972588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:48.972562 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:49.104392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.104299 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr\" (UniqueName: \"kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr\") pod \"cbcc03fd-9fa9-41f4-a46c-48e637963102\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " Nov 03 20:47:49.104545 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.104526 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca\") pod \"cbcc03fd-9fa9-41f4-a46c-48e637963102\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " Nov 03 20:47:49.104633 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.104578 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca\") pod \"cbcc03fd-9fa9-41f4-a46c-48e637963102\" (UID: \"cbcc03fd-9fa9-41f4-a46c-48e637963102\") " Nov 03 20:47:49.104886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.104840 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca" pod "cbcc03fd-9fa9-41f4-a46c-48e637963102" (UID: "cbcc03fd-9fa9-41f4-a46c-48e637963102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:47:49.105076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.105060 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.106604 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.106577 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca" pod "cbcc03fd-9fa9-41f4-a46c-48e637963102" (UID: "cbcc03fd-9fa9-41f4-a46c-48e637963102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:47:49.106887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.106862 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr" pod "cbcc03fd-9fa9-41f4-a46c-48e637963102" (UID: "cbcc03fd-9fa9-41f4-a46c-48e637963102"). InnerVolumeSpecName "kube-api-access-pszzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:47:49.206251 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.206220 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbcc03fd-9fa9-41f4-a46c-48e637963102-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.206251 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.206248 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pszzr\" (UniqueName: \"kubernetes.io/projected/cbcc03fd-9fa9-41f4-a46c-48e637963102-kube-api-access-pszzr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:47:49.821940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.821915 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" Nov 03 20:47:49.821940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.821932 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g" event={"ID":"cbcc03fd-9fa9-41f4-a46c-48e637963102","Type":"ContainerDied","Data":"af7bcb104ec3d2c29fd0e9ec2e35279d323a59c67fdc37ec8ce5002c23884442"} Nov 03 20:47:49.822148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:49.821973 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af7bcb104ec3d2c29fd0e9ec2e35279d323a59c67fdc37ec8ce5002c23884442" Nov 03 20:47:54.265699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.265669 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p"] Nov 03 20:47:54.266298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.266285 5932 cpu_manager.go:414] "containerMap: removing container" podUID="cbcc03fd-9fa9-41f4-a46c-48e637963102" containerName="revoke-keys" Nov 03 20:47:54.266353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.266300 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcc03fd-9fa9-41f4-a46c-48e637963102" containerName="revoke-keys" Nov 03 20:47:54.266417 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.266408 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcc03fd-9fa9-41f4-a46c-48e637963102" containerName="revoke-keys" Nov 03 20:47:54.271148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.271126 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.339323 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.339285 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p"] Nov 03 20:47:54.368209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368179 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368223 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368247 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368350 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368395 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368458 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368613 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368665 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgq8\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.368839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.368819 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470077 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470110 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470158 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470195 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470225 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470277 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470296 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgq8\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470366 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470419 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470638 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470652 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.470912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470742 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.471000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.470983 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.471041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.471012 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.472767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.472741 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.472929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.472742 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.527173 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.527122 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgq8\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.527463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.527446 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:54.600732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:54.595730 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:55.061698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:55.061597 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p"] Nov 03 20:47:55.064155 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:47:55.064130 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d854bc_b23f_4902_9985_ded9777dcf0a.slice/crio-68e84b6b48a7eb4f1c256a21c8383d6544623d423e1295710e41590f25769c5a WatchSource:0}: Error finding container 68e84b6b48a7eb4f1c256a21c8383d6544623d423e1295710e41590f25769c5a: Status 404 returned error can't find the container with id 68e84b6b48a7eb4f1c256a21c8383d6544623d423e1295710e41590f25769c5a Nov 03 20:47:55.066667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:55.066605 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:47:55.066747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:55.066721 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:47:55.066784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:55.066753 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:47:55.873347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:55.873313 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" event={"ID":"f4d854bc-b23f-4902-9985-ded9777dcf0a","Type":"ContainerStarted","Data":"6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c"} Nov 03 20:47:55.875787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:55.873403 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" event={"ID":"f4d854bc-b23f-4902-9985-ded9777dcf0a","Type":"ContainerStarted","Data":"68e84b6b48a7eb4f1c256a21c8383d6544623d423e1295710e41590f25769c5a"} Nov 03 20:47:56.596730 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:56.596695 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:47:56.598438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:56.598413 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:47:56.598525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:56.598470 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:47:57.596096 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:57.596059 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:47:57.596471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:57.596118 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:47:58.596768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:58.596733 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:47:58.597147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:58.596796 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:47:59.596654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:59.596595 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:47:59.596843 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:47:59.596668 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:48:00.596716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:00.596684 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:48:00.596894 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:00.596756 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:48:01.596441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:01.596401 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Nov 03 20:48:01.596683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:01.596473 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Nov 03 20:48:02.600893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:02.600867 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:48:02.601291 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:02.601258 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:48:02.601945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:02.601925 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:48:02.761489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:02.761396 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podStartSLOduration=8.761381803 podStartE2EDuration="8.761381803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:56.09797717 +0000 UTC m=+1084.163218645" watchObservedRunningTime="2026-11-03 20:48:02.761381803 +0000 UTC m=+1090.826623276" Nov 03 20:48:14.152090 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:48:14.152052 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:48:39.853936 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.853904 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4"] Nov 03 20:48:39.858199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.858182 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.958993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.958958 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4"] Nov 03 20:48:39.980755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.980714 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.980946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.980788 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.980946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.980885 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.980946 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.980939 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.981099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.981050 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.981184 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.981167 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.981251 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.981219 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.981306 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.981251 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:39.981306 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:39.981283 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f7jk\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082300 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082338 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082359 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f7jk\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082420 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082451 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082496 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082554 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082649 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.082801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.082739 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.083070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.083050 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.083178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.083091 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.083178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.083163 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.083263 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.083225 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.083412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.083395 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.085269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.085243 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.085399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.085381 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.164301 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.164230 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f7jk\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.180326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.180259 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:40.473028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:40.472944 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:41.019608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.019434 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4"] Nov 03 20:48:41.032425 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.032386 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:48:41.032530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.032496 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:48:41.032600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.032573 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:48:41.295815 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.295723 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" event={"ID":"d768da6e-fcd6-4389-babb-f662d3763675","Type":"ContainerStarted","Data":"42564e5a06712a7e210bbacc7c689318084dbab6474c7ea892f821b9a9bf66fc"} Nov 03 20:48:41.295815 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.295803 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" event={"ID":"d768da6e-fcd6-4389-babb-f662d3763675","Type":"ContainerStarted","Data":"0fad13ebcca4ceaf598de82b253635b466db7797a84fe53701eb2787b2311c9a"} Nov 03 20:48:41.415204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.415156 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podStartSLOduration=2.415141238 podStartE2EDuration="2.415141238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:48:41.413816322 +0000 UTC m=+1129.479057795" watchObservedRunningTime="2026-11-03 20:48:41.415141238 +0000 UTC m=+1129.480382760" Nov 03 20:48:41.474668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:41.474425 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:42.476154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:42.476112 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:48:42.476775 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:42.476733 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:48:43.473852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:43.473821 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:48:43.474036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:43.473866 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:48:44.173021 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:48:44.172982 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:48:44.476855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:44.476761 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:48:44.476855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:44.476815 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:48:45.474168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:45.474135 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:48:45.474568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:45.474200 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:48:46.474882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:46.474845 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:48:46.475314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:46.474916 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:48:47.474225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:47.474189 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Nov 03 20:48:47.474435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:47.474251 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Nov 03 20:48:48.478127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:48.478092 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:48.478589 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:48.478550 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:48:48.479185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:48:48.479165 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 20:49:08.534751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.534718 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2"] Nov 03 20:49:08.540089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.540070 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.600327 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.600299 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nsh\" (UniqueName: \"kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.600506 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.600365 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.600581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.600525 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.600581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.600553 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.600750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.600676 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.600750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.600709 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.636964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.636937 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"distinct-w1-0cd49b-kserve-self-signed-certs\"" Nov 03 20:49:08.640569 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.640546 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2"] Nov 03 20:49:08.705267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.705065 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nsh\" (UniqueName: \"kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.705454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.705354 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.705523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.705500 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.705579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.705530 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.709208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.705601 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.709208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.705692 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.709208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.706241 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.710077 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.710043 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.710931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.710907 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.716943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.716886 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.722616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.722246 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.767145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.767122 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nsh\" (UniqueName: \"kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh\") pod \"distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:08.852695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:08.852613 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:09.235969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:09.235931 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2"] Nov 03 20:49:09.240554 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:49:09.240525 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a379423_fe9d_4b10_b62a_1a9b5442cd4a.slice/crio-70f78dabc482bc18a97f4c225df2f47c1e246c1ac30ba4d275d4fbd2f839fa41 WatchSource:0}: Error finding container 70f78dabc482bc18a97f4c225df2f47c1e246c1ac30ba4d275d4fbd2f839fa41: Status 404 returned error can't find the container with id 70f78dabc482bc18a97f4c225df2f47c1e246c1ac30ba4d275d4fbd2f839fa41 Nov 03 20:49:09.601616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:09.601525 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" event={"ID":"0a379423-fe9d-4b10-b62a-1a9b5442cd4a","Type":"ContainerStarted","Data":"f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7"} Nov 03 20:49:09.601616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:09.601569 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" event={"ID":"0a379423-fe9d-4b10-b62a-1a9b5442cd4a","Type":"ContainerStarted","Data":"70f78dabc482bc18a97f4c225df2f47c1e246c1ac30ba4d275d4fbd2f839fa41"} Nov 03 20:49:09.717442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:09.717378 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podStartSLOduration=1.717359619 podStartE2EDuration="1.717359619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:49:09.710699988 +0000 UTC m=+1157.775941460" watchObservedRunningTime="2026-11-03 20:49:09.717359619 +0000 UTC m=+1157.782601095" Nov 03 20:49:09.861553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:09.861461 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:11.618511 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:11.618432 5932 generic.go:356] "Generic (PLEG): container finished" podID="ae58f20c-eff4-47e6-b351-4a8bc416849b" containerID="63185253cfdfca982d43dd8738b9122d7172dd3e5760fb93760ac7b3c1ffcf5b" exitCode=0 Nov 03 20:49:11.618931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:11.618504 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" event={"ID":"ae58f20c-eff4-47e6-b351-4a8bc416849b","Type":"ContainerDied","Data":"63185253cfdfca982d43dd8738b9122d7172dd3e5760fb93760ac7b3c1ffcf5b"} Nov 03 20:49:12.786682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.786656 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:49:12.864312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864282 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle\") pod \"ae58f20c-eff4-47e6-b351-4a8bc416849b\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " Nov 03 20:49:12.864450 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864404 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle\") pod \"ae58f20c-eff4-47e6-b351-4a8bc416849b\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " Nov 03 20:49:12.864450 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864430 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk\" (UniqueName: \"kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk\") pod \"ae58f20c-eff4-47e6-b351-4a8bc416849b\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " Nov 03 20:49:12.864535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864505 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path\") pod \"ae58f20c-eff4-47e6-b351-4a8bc416849b\" (UID: \"ae58f20c-eff4-47e6-b351-4a8bc416849b\") " Nov 03 20:49:12.864789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864762 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle" pod "ae58f20c-eff4-47e6-b351-4a8bc416849b" (UID: "ae58f20c-eff4-47e6-b351-4a8bc416849b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:12.864789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864779 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle" pod "ae58f20c-eff4-47e6-b351-4a8bc416849b" (UID: "ae58f20c-eff4-47e6-b351-4a8bc416849b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:12.864945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864865 5932 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-trusted-ca-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:12.864945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864879 5932 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae58f20c-eff4-47e6-b351-4a8bc416849b-service-ca-bundle\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:12.865037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.864953 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path" pod "ae58f20c-eff4-47e6-b351-4a8bc416849b" (UID: "ae58f20c-eff4-47e6-b351-4a8bc416849b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:12.866593 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.866567 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk" pod "ae58f20c-eff4-47e6-b351-4a8bc416849b" (UID: "ae58f20c-eff4-47e6-b351-4a8bc416849b"). InnerVolumeSpecName "kube-api-access-5kjlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:12.966301 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.966235 5932 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ae58f20c-eff4-47e6-b351-4a8bc416849b-archives-path\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:12.966301 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:12.966258 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kjlk\" (UniqueName: \"kubernetes.io/projected/ae58f20c-eff4-47e6-b351-4a8bc416849b-kube-api-access-5kjlk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:13.641574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:13.641539 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" Nov 03 20:49:13.641779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:13.641575 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-ggs45-tf5xz" event={"ID":"ae58f20c-eff4-47e6-b351-4a8bc416849b","Type":"ContainerDied","Data":"1a94ef18cf05fbb46b4d7101df7f8dfb33fde3cd6c2acd80712367f9a6ba9d0f"} Nov 03 20:49:13.641779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:13.641680 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a94ef18cf05fbb46b4d7101df7f8dfb33fde3cd6c2acd80712367f9a6ba9d0f" Nov 03 20:49:14.151169 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:49:14.151129 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:49:18.853510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:18.853475 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:18.865683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:18.865657 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:19.701028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:19.700993 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:49:44.149991 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:49:44.149915 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:49:52.292094 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:52.292067 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:49:52.294314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:52.294292 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:49:54.231313 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.231276 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z"] Nov 03 20:49:54.231961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.231945 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ae58f20c-eff4-47e6-b351-4a8bc416849b" containerName="insights-gathering" Nov 03 20:49:54.232011 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.231962 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae58f20c-eff4-47e6-b351-4a8bc416849b" containerName="insights-gathering" Nov 03 20:49:54.232088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.232078 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae58f20c-eff4-47e6-b351-4a8bc416849b" containerName="insights-gathering" Nov 03 20:49:54.235949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.235924 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.289949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.289925 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z"] Nov 03 20:49:54.330898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.330875 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:49:54.331035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.330916 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:49:54.358781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.358756 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.358909 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.358791 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz2f5\" (UniqueName: \"kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.358978 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.358941 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.460735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.460705 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.460878 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.460740 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz2f5\" (UniqueName: \"kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.460878 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.460806 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.461515 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.461495 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.463236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.463212 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.517882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.517826 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz2f5\" (UniqueName: \"kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5\") pod \"maas-api-revoke-sub-236db804-ccl5z\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.549323 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.549300 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:54.925291 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.925250 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z"] Nov 03 20:49:54.930307 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:49:54.930274 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67f0a98_41c3_4222_af2d_99d5a986f4df.slice/crio-fc2dbd33d1c30e9dd16bff0f1cbc273bcfd7113341c5c8254d02c07ad942be5a WatchSource:0}: Error finding container fc2dbd33d1c30e9dd16bff0f1cbc273bcfd7113341c5c8254d02c07ad942be5a: Status 404 returned error can't find the container with id fc2dbd33d1c30e9dd16bff0f1cbc273bcfd7113341c5c8254d02c07ad942be5a Nov 03 20:49:54.959712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:54.959685 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" event={"ID":"e67f0a98-41c3-4222-af2d-99d5a986f4df","Type":"ContainerStarted","Data":"fc2dbd33d1c30e9dd16bff0f1cbc273bcfd7113341c5c8254d02c07ad942be5a"} Nov 03 20:49:55.968103 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:55.968074 5932 generic.go:356] "Generic (PLEG): container finished" podID="e67f0a98-41c3-4222-af2d-99d5a986f4df" containerID="bd7e377306175fa1bc658950c7c85af0cc8f6d985513d2b1c4ca9f6018d0dcee" exitCode=0 Nov 03 20:49:55.968426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:55.968162 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" event={"ID":"e67f0a98-41c3-4222-af2d-99d5a986f4df","Type":"ContainerDied","Data":"bd7e377306175fa1bc658950c7c85af0cc8f6d985513d2b1c4ca9f6018d0dcee"} Nov 03 20:49:57.115785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.115760 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:57.295553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.295471 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5\" (UniqueName: \"kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5\") pod \"e67f0a98-41c3-4222-af2d-99d5a986f4df\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " Nov 03 20:49:57.295755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.295710 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca\") pod \"e67f0a98-41c3-4222-af2d-99d5a986f4df\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " Nov 03 20:49:57.295826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.295776 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca\") pod \"e67f0a98-41c3-4222-af2d-99d5a986f4df\" (UID: \"e67f0a98-41c3-4222-af2d-99d5a986f4df\") " Nov 03 20:49:57.296052 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.296024 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca" pod "e67f0a98-41c3-4222-af2d-99d5a986f4df" (UID: "e67f0a98-41c3-4222-af2d-99d5a986f4df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:49:57.296261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.296243 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:57.297455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.297431 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5" pod "e67f0a98-41c3-4222-af2d-99d5a986f4df" (UID: "e67f0a98-41c3-4222-af2d-99d5a986f4df"). InnerVolumeSpecName "kube-api-access-sz2f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:57.297678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.297663 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca" pod "e67f0a98-41c3-4222-af2d-99d5a986f4df" (UID: "e67f0a98-41c3-4222-af2d-99d5a986f4df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:49:57.397532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.397506 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e67f0a98-41c3-4222-af2d-99d5a986f4df-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:57.397532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.397532 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz2f5\" (UniqueName: \"kubernetes.io/projected/e67f0a98-41c3-4222-af2d-99d5a986f4df-kube-api-access-sz2f5\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:57.983741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.983711 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" Nov 03 20:49:57.983925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.983744 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z" event={"ID":"e67f0a98-41c3-4222-af2d-99d5a986f4df","Type":"ContainerDied","Data":"fc2dbd33d1c30e9dd16bff0f1cbc273bcfd7113341c5c8254d02c07ad942be5a"} Nov 03 20:49:57.983925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:49:57.983814 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2dbd33d1c30e9dd16bff0f1cbc273bcfd7113341c5c8254d02c07ad942be5a" Nov 03 20:50:14.151228 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:50:14.151188 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:50:22.849689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.849647 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8"] Nov 03 20:50:22.850287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.850272 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e67f0a98-41c3-4222-af2d-99d5a986f4df" containerName="revoke-keys" Nov 03 20:50:22.850287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.850288 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67f0a98-41c3-4222-af2d-99d5a986f4df" containerName="revoke-keys" Nov 03 20:50:22.850393 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.850383 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67f0a98-41c3-4222-af2d-99d5a986f4df" containerName="revoke-keys" Nov 03 20:50:22.854014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.853998 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:22.949474 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.949436 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-2a5ec9\"/\"distinct-2-w5-2a5ec9-kserve-self-signed-certs\"" Nov 03 20:50:22.953445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:22.953413 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8"] Nov 03 20:50:23.041953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.041917 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.042169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.042006 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.042169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.042032 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.042169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.042119 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.042330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.042200 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.042330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.042311 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8j9\" (UniqueName: \"kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143069 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143026 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8j9\" (UniqueName: \"kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143095 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143135 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143151 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143245 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143589 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143713 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.143924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.143837 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.145655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.145606 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.145843 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.145825 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.200383 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.200346 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8j9\" (UniqueName: \"kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9\") pod \"distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.467046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.466957 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:23.851651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:23.851533 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8"] Nov 03 20:50:23.856143 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:50:23.856114 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02a51e7f_7796_446d_a553_bb712200a85a.slice/crio-bac9e5cdaa7c0be3f425a3c88588ff4edab950f6429f619b50154b2e403eeb04 WatchSource:0}: Error finding container bac9e5cdaa7c0be3f425a3c88588ff4edab950f6429f619b50154b2e403eeb04: Status 404 returned error can't find the container with id bac9e5cdaa7c0be3f425a3c88588ff4edab950f6429f619b50154b2e403eeb04 Nov 03 20:50:24.200284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:24.200249 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" event={"ID":"02a51e7f-7796-446d-a553-bb712200a85a","Type":"ContainerStarted","Data":"012ec6e1578a8dd721e79500db25289d0ce7ef630ea929e76a559d246b97a93b"} Nov 03 20:50:24.200456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:24.200295 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" event={"ID":"02a51e7f-7796-446d-a553-bb712200a85a","Type":"ContainerStarted","Data":"bac9e5cdaa7c0be3f425a3c88588ff4edab950f6429f619b50154b2e403eeb04"} Nov 03 20:50:24.309977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:24.309924 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podStartSLOduration=2.30991037 podStartE2EDuration="2.30991037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:24.309144554 +0000 UTC m=+1232.374386040" watchObservedRunningTime="2026-11-03 20:50:24.30991037 +0000 UTC m=+1232.375151843" Nov 03 20:50:24.467822 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:24.467739 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:33.467098 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:33.467065 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:33.479452 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:33.479424 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:34.286861 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:34.286828 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:50:34.858902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:34.858865 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85"] Nov 03 20:50:34.864650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:34.864600 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:34.913800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:34.913767 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:34.914745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:34.914717 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85"] Nov 03 20:50:34.955334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:34.955295 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:50:35.008032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.007994 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.008394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.008070 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.008394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.008261 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85496\" (UniqueName: \"kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.109022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.108939 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85496\" (UniqueName: \"kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.109022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.108984 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.109227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.109040 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.109853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.109826 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.111518 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.111491 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.166963 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.166927 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85496\" (UniqueName: \"kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496\") pod \"maas-api-revoke-sub-c955f2ea-87f85\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.181741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.181716 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:35.559210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:35.559172 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85"] Nov 03 20:50:35.563289 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:50:35.563260 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda819c87e_860b_4538_8918_d8b6515cbba8.slice/crio-4bb7c45383eabf4b0e8c9a6a4ccefaedcbd7bc5a526348e4f08d0a972ad0e12c WatchSource:0}: Error finding container 4bb7c45383eabf4b0e8c9a6a4ccefaedcbd7bc5a526348e4f08d0a972ad0e12c: Status 404 returned error can't find the container with id 4bb7c45383eabf4b0e8c9a6a4ccefaedcbd7bc5a526348e4f08d0a972ad0e12c Nov 03 20:50:36.295409 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:36.295367 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" event={"ID":"a819c87e-860b-4538-8918-d8b6515cbba8","Type":"ContainerStarted","Data":"6488878d3b9a48aa4e7e960b854cf947776688f44c29c52506353d44f780ce07"} Nov 03 20:50:36.295842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:36.295435 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" event={"ID":"a819c87e-860b-4538-8918-d8b6515cbba8","Type":"ContainerStarted","Data":"4bb7c45383eabf4b0e8c9a6a4ccefaedcbd7bc5a526348e4f08d0a972ad0e12c"} Nov 03 20:50:36.405286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:36.405237 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" podStartSLOduration=2.405223595 podStartE2EDuration="2.405223595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:36.403835896 +0000 UTC m=+1244.469077370" watchObservedRunningTime="2026-11-03 20:50:36.405223595 +0000 UTC m=+1244.470465068" Nov 03 20:50:37.332178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:37.332140 5932 generic.go:356] "Generic (PLEG): container finished" podID="a819c87e-860b-4538-8918-d8b6515cbba8" containerID="6488878d3b9a48aa4e7e960b854cf947776688f44c29c52506353d44f780ce07" exitCode=0 Nov 03 20:50:37.332709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:37.332229 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" event={"ID":"a819c87e-860b-4538-8918-d8b6515cbba8","Type":"ContainerDied","Data":"6488878d3b9a48aa4e7e960b854cf947776688f44c29c52506353d44f780ce07"} Nov 03 20:50:38.543044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.543018 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:38.667112 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.667081 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca\") pod \"a819c87e-860b-4538-8918-d8b6515cbba8\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " Nov 03 20:50:38.667295 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.667176 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496\" (UniqueName: \"kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496\") pod \"a819c87e-860b-4538-8918-d8b6515cbba8\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " Nov 03 20:50:38.667295 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.667220 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca\") pod \"a819c87e-860b-4538-8918-d8b6515cbba8\" (UID: \"a819c87e-860b-4538-8918-d8b6515cbba8\") " Nov 03 20:50:38.667600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.667569 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca" pod "a819c87e-860b-4538-8918-d8b6515cbba8" (UID: "a819c87e-860b-4538-8918-d8b6515cbba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:38.667819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.667793 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:38.669235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.669215 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca" pod "a819c87e-860b-4538-8918-d8b6515cbba8" (UID: "a819c87e-860b-4538-8918-d8b6515cbba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:38.669395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.669371 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496" pod "a819c87e-860b-4538-8918-d8b6515cbba8" (UID: "a819c87e-860b-4538-8918-d8b6515cbba8"). InnerVolumeSpecName "kube-api-access-85496". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:38.772438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.772366 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a819c87e-860b-4538-8918-d8b6515cbba8-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:38.772438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:38.772407 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85496\" (UniqueName: \"kubernetes.io/projected/a819c87e-860b-4538-8918-d8b6515cbba8-kube-api-access-85496\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:39.363120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:39.363088 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" Nov 03 20:50:39.363293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:39.363119 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85" event={"ID":"a819c87e-860b-4538-8918-d8b6515cbba8","Type":"ContainerDied","Data":"4bb7c45383eabf4b0e8c9a6a4ccefaedcbd7bc5a526348e4f08d0a972ad0e12c"} Nov 03 20:50:39.363293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:39.363168 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bb7c45383eabf4b0e8c9a6a4ccefaedcbd7bc5a526348e4f08d0a972ad0e12c" Nov 03 20:50:40.511825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.511790 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl"] Nov 03 20:50:40.512386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.512373 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a819c87e-860b-4538-8918-d8b6515cbba8" containerName="revoke-keys" Nov 03 20:50:40.512431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.512387 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a819c87e-860b-4538-8918-d8b6515cbba8" containerName="revoke-keys" Nov 03 20:50:40.512517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.512508 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a819c87e-860b-4538-8918-d8b6515cbba8" containerName="revoke-keys" Nov 03 20:50:40.516153 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.516132 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.591024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.590984 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.591201 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.591104 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.591201 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.591157 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.591201 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.591192 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.591429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.591394 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqw87\" (UniqueName: \"kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.591608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.591481 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.611679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.611610 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"openshift-service-ca.crt\"" Nov 03 20:50:40.611679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.611670 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"kube-root-ca.crt\"" Nov 03 20:50:40.611888 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.611755 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"distinct-2-w3-88228a-kserve-self-signed-certs\"" Nov 03 20:50:40.611888 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.611816 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-88228a\"/\"default-dockercfg-8xb4k\"" Nov 03 20:50:40.615807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.615783 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl"] Nov 03 20:50:40.692659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.692604 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.692955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.692933 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.693088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.693073 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.693271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.693252 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqw87\" (UniqueName: \"kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.693423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.693408 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.693580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.693562 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.694164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.694138 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.695347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.695046 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.695347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.695302 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.723716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.706187 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.723716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.723594 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.751937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.751908 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqw87\" (UniqueName: \"kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87\") pod \"distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:40.829246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:40.829164 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:41.050937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.050902 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956"] Nov 03 20:50:41.054851 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.054829 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41b2b3b4-l8956"] Nov 03 20:50:41.206565 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.206534 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl"] Nov 03 20:50:41.211145 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:50:41.211116 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd677dd53_9912_4d03_8a81_7174bf5f9b88.slice/crio-d09d1b512970d7eb882026446bd0f4a34c7029322d7801c022d39f56e11e587c WatchSource:0}: Error finding container d09d1b512970d7eb882026446bd0f4a34c7029322d7801c022d39f56e11e587c: Status 404 returned error can't find the container with id d09d1b512970d7eb882026446bd0f4a34c7029322d7801c022d39f56e11e587c Nov 03 20:50:41.380290 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.380256 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" event={"ID":"d677dd53-9912-4d03-8a81-7174bf5f9b88","Type":"ContainerStarted","Data":"131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c"} Nov 03 20:50:41.380461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.380299 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" event={"ID":"d677dd53-9912-4d03-8a81-7174bf5f9b88","Type":"ContainerStarted","Data":"d09d1b512970d7eb882026446bd0f4a34c7029322d7801c022d39f56e11e587c"} Nov 03 20:50:41.491180 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.491079 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podStartSLOduration=1.49106469 podStartE2EDuration="1.49106469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:41.489710914 +0000 UTC m=+1249.554952390" watchObservedRunningTime="2026-11-03 20:50:41.49106469 +0000 UTC m=+1249.556306162" Nov 03 20:50:41.830205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:41.830119 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:42.052493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:42.052457 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d36cb217-8eaa-47b8-b1d4-bd975f17a548" path="/var/lib/kubelet/pods/d36cb217-8eaa-47b8-b1d4-bd975f17a548/volumes" Nov 03 20:50:44.150958 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:50:44.150912 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:50:47.046950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:47.046920 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp"] Nov 03 20:50:47.092913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:47.092878 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d747c76-ndpxp"] Nov 03 20:50:48.053281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:48.053237 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e22d4c-2e1f-494f-96c2-984353adbaf5" path="/var/lib/kubelet/pods/87e22d4c-2e1f-494f-96c2-984353adbaf5/volumes" Nov 03 20:50:50.461047 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.461013 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf"] Nov 03 20:50:50.466384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.466361 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.518712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.518680 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf"] Nov 03 20:50:50.559906 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.559870 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:50:50.560071 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.559989 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:50:50.627419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.627374 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzzjk\" (UniqueName: \"kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.627791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.627528 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.627791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.627581 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.728739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.728655 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.728739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.728714 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.728956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.728880 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzzjk\" (UniqueName: \"kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.729306 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.729283 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.731222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.731202 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.786441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.786406 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzzjk\" (UniqueName: \"kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk\") pod \"maas-api-revoke-sub-ae926a78-kbtpf\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:50.830211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.830173 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:50.843325 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:50.843298 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:51.081049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:51.080968 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:51.457277 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:51.457247 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf"] Nov 03 20:50:51.470384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:51.470358 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:50:52.059308 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:52.059215 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5"] Nov 03 20:50:52.087006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:52.086960 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6158410-wrlc5"] Nov 03 20:50:52.466069 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:52.466035 5932 generic.go:356] "Generic (PLEG): container finished" podID="c3b1d972-1316-4093-8f3a-d090d52992e9" containerID="022d5350fc5654ce2564df7c40c557d3b4398df329ea03477ef70b7f51a5e518" exitCode=0 Nov 03 20:50:52.466250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:52.466106 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" event={"ID":"c3b1d972-1316-4093-8f3a-d090d52992e9","Type":"ContainerDied","Data":"022d5350fc5654ce2564df7c40c557d3b4398df329ea03477ef70b7f51a5e518"} Nov 03 20:50:52.466250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:52.466151 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" event={"ID":"c3b1d972-1316-4093-8f3a-d090d52992e9","Type":"ContainerStarted","Data":"929eb01f9cbb3203dfbf579fd243b16b65197af22c351d0773362e5030e0df0b"} Nov 03 20:50:53.620557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.620532 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:53.656319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.656287 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca\") pod \"c3b1d972-1316-4093-8f3a-d090d52992e9\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " Nov 03 20:50:53.658281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.658256 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca" pod "c3b1d972-1316-4093-8f3a-d090d52992e9" (UID: "c3b1d972-1316-4093-8f3a-d090d52992e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:50:53.757001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.756916 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca\") pod \"c3b1d972-1316-4093-8f3a-d090d52992e9\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " Nov 03 20:50:53.757001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.756963 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk\" (UniqueName: \"kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk\") pod \"c3b1d972-1316-4093-8f3a-d090d52992e9\" (UID: \"c3b1d972-1316-4093-8f3a-d090d52992e9\") " Nov 03 20:50:53.757335 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.757309 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca" pod "c3b1d972-1316-4093-8f3a-d090d52992e9" (UID: "c3b1d972-1316-4093-8f3a-d090d52992e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:50:53.757569 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.757554 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:53.757660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.757576 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3b1d972-1316-4093-8f3a-d090d52992e9-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:53.759532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.759499 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk" pod "c3b1d972-1316-4093-8f3a-d090d52992e9" (UID: "c3b1d972-1316-4093-8f3a-d090d52992e9"). InnerVolumeSpecName "kube-api-access-qzzjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:50:53.858536 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:53.858501 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzzjk\" (UniqueName: \"kubernetes.io/projected/c3b1d972-1316-4093-8f3a-d090d52992e9-kube-api-access-qzzjk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:50:54.052501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:54.052426 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f2c3c9-d951-4423-8e0c-f63266099584" path="/var/lib/kubelet/pods/71f2c3c9-d951-4423-8e0c-f63266099584/volumes" Nov 03 20:50:54.483198 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:54.483166 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" Nov 03 20:50:54.483376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:54.483195 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf" event={"ID":"c3b1d972-1316-4093-8f3a-d090d52992e9","Type":"ContainerDied","Data":"929eb01f9cbb3203dfbf579fd243b16b65197af22c351d0773362e5030e0df0b"} Nov 03 20:50:54.483376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:50:54.483244 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="929eb01f9cbb3203dfbf579fd243b16b65197af22c351d0773362e5030e0df0b" Nov 03 20:51:04.698127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:04.697590 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:51:09.012023 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.011989 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl"] Nov 03 20:51:09.012601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.012580 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c3b1d972-1316-4093-8f3a-d090d52992e9" containerName="revoke-keys" Nov 03 20:51:09.012684 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.012602 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b1d972-1316-4093-8f3a-d090d52992e9" containerName="revoke-keys" Nov 03 20:51:09.012763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.012753 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b1d972-1316-4093-8f3a-d090d52992e9" containerName="revoke-keys" Nov 03 20:51:09.018133 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.018111 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.070528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.070494 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl"] Nov 03 20:51:09.076883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.076852 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.077192 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.077164 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d955\" (UniqueName: \"kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.077562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.077448 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.111832 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.111801 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:51:09.111832 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.111817 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:09.178800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.178761 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d955\" (UniqueName: \"kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.179000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.178847 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.179000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.178882 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.179693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.179670 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.181288 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.181260 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.236751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.236721 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d955\" (UniqueName: \"kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955\") pod \"maas-api-revoke-sub-1bc92735-kb9zl\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.332945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.332858 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:09.715798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:09.715763 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl"] Nov 03 20:51:09.719834 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:09.719803 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0faac424_a52e_46d1_9c66_f06f58b5c421.slice/crio-a215e3b2a5670122c4bdee6b6d2e92a9ce1be7665251d03167c26b9ce624c504 WatchSource:0}: Error finding container a215e3b2a5670122c4bdee6b6d2e92a9ce1be7665251d03167c26b9ce624c504: Status 404 returned error can't find the container with id a215e3b2a5670122c4bdee6b6d2e92a9ce1be7665251d03167c26b9ce624c504 Nov 03 20:51:10.686821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:10.686786 5932 generic.go:356] "Generic (PLEG): container finished" podID="0faac424-a52e-46d1-9c66-f06f58b5c421" containerID="7d8fca06b7f2c9d8bcbd167bc42253f657f818646897caf11300e52955cbf217" exitCode=0 Nov 03 20:51:10.687211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:10.686869 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" event={"ID":"0faac424-a52e-46d1-9c66-f06f58b5c421","Type":"ContainerDied","Data":"7d8fca06b7f2c9d8bcbd167bc42253f657f818646897caf11300e52955cbf217"} Nov 03 20:51:10.687211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:10.686918 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" event={"ID":"0faac424-a52e-46d1-9c66-f06f58b5c421","Type":"ContainerStarted","Data":"a215e3b2a5670122c4bdee6b6d2e92a9ce1be7665251d03167c26b9ce624c504"} Nov 03 20:51:11.846947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.846922 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:11.913468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.913437 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca\") pod \"0faac424-a52e-46d1-9c66-f06f58b5c421\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " Nov 03 20:51:11.913645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.913570 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca\") pod \"0faac424-a52e-46d1-9c66-f06f58b5c421\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " Nov 03 20:51:11.913726 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.913708 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955\" (UniqueName: \"kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955\") pod \"0faac424-a52e-46d1-9c66-f06f58b5c421\" (UID: \"0faac424-a52e-46d1-9c66-f06f58b5c421\") " Nov 03 20:51:11.913993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.913966 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca" pod "0faac424-a52e-46d1-9c66-f06f58b5c421" (UID: "0faac424-a52e-46d1-9c66-f06f58b5c421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:11.914170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.914152 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:11.915609 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.915582 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca" pod "0faac424-a52e-46d1-9c66-f06f58b5c421" (UID: "0faac424-a52e-46d1-9c66-f06f58b5c421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:11.915806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:11.915786 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955" pod "0faac424-a52e-46d1-9c66-f06f58b5c421" (UID: "0faac424-a52e-46d1-9c66-f06f58b5c421"). InnerVolumeSpecName "kube-api-access-2d955". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:12.015244 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:12.015164 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d955\" (UniqueName: \"kubernetes.io/projected/0faac424-a52e-46d1-9c66-f06f58b5c421-kube-api-access-2d955\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:12.015244 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:12.015192 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0faac424-a52e-46d1-9c66-f06f58b5c421-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:12.710687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:12.710656 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" Nov 03 20:51:12.710687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:12.710668 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl" event={"ID":"0faac424-a52e-46d1-9c66-f06f58b5c421","Type":"ContainerDied","Data":"a215e3b2a5670122c4bdee6b6d2e92a9ce1be7665251d03167c26b9ce624c504"} Nov 03 20:51:12.710961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:12.710750 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a215e3b2a5670122c4bdee6b6d2e92a9ce1be7665251d03167c26b9ce624c504" Nov 03 20:51:13.720041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.720008 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf"] Nov 03 20:51:13.720600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.720586 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0faac424-a52e-46d1-9c66-f06f58b5c421" containerName="revoke-keys" Nov 03 20:51:13.720669 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.720601 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0faac424-a52e-46d1-9c66-f06f58b5c421" containerName="revoke-keys" Nov 03 20:51:13.720759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.720748 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0faac424-a52e-46d1-9c66-f06f58b5c421" containerName="revoke-keys" Nov 03 20:51:13.724602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.724580 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.820043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.820014 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0cd49b\"/\"distinct-2-w1-0cd49b-kserve-self-signed-certs\"" Nov 03 20:51:13.824309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.824277 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf"] Nov 03 20:51:13.836331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.836290 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfp6\" (UniqueName: \"kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.836491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.836337 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.836491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.836462 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.836604 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.836510 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.836604 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.836585 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.836713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.836682 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938374 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfp6\" (UniqueName: \"kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938589 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938450 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938584 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938670 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938695 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938769 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938779 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.938896 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.938879 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.939092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.939070 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.940923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.940904 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.941219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.941200 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:13.995883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:13.995809 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfp6\" (UniqueName: \"kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6\") pod \"distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:14.037867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.037832 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:14.234132 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:51:14.234090 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:51:14.418160 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.418124 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf"] Nov 03 20:51:14.422102 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:14.422075 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod692a7c2e_4cbf_407c_b821_aeb0582d9e32.slice/crio-3967f5c8911a3ad21c789b82f22244f6927fa93a26a519b9232f7e94ac6cf091 WatchSource:0}: Error finding container 3967f5c8911a3ad21c789b82f22244f6927fa93a26a519b9232f7e94ac6cf091: Status 404 returned error can't find the container with id 3967f5c8911a3ad21c789b82f22244f6927fa93a26a519b9232f7e94ac6cf091 Nov 03 20:51:14.434166 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.434145 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6"] Nov 03 20:51:14.439969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.439939 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.488791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.488760 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:14.488953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.488875 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:51:14.523354 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.522885 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6"] Nov 03 20:51:14.555579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.555538 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8cc\" (UniqueName: \"kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.555739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.555675 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.555821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.555805 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.656948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.656907 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.657119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.656965 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.657204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.657116 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8cc\" (UniqueName: \"kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.657735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.657709 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.659385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.659367 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.714536 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.714452 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8cc\" (UniqueName: \"kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc\") pod \"maas-api-revoke-sub-f8234aaa-d26c6\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.728932 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.728899 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" event={"ID":"692a7c2e-4cbf-407c-b821-aeb0582d9e32","Type":"ContainerStarted","Data":"897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1"} Nov 03 20:51:14.729316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.728941 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" event={"ID":"692a7c2e-4cbf-407c-b821-aeb0582d9e32","Type":"ContainerStarted","Data":"3967f5c8911a3ad21c789b82f22244f6927fa93a26a519b9232f7e94ac6cf091"} Nov 03 20:51:14.831178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.831149 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:14.846394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:14.846307 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podStartSLOduration=1.846291773 podStartE2EDuration="1.846291773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:14.841727141 +0000 UTC m=+1282.906968615" watchObservedRunningTime="2026-11-03 20:51:14.846291773 +0000 UTC m=+1282.911533268" Nov 03 20:51:15.038930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:15.038840 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:15.209188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:15.209152 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6"] Nov 03 20:51:15.213558 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:15.213535 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1081749c_9383_4400_871d_e47f523adb92.slice/crio-df80e094cdc17337426660f6b4a38f19c5a1fc9fe412031a6a80a38a0b30fad4 WatchSource:0}: Error finding container df80e094cdc17337426660f6b4a38f19c5a1fc9fe412031a6a80a38a0b30fad4: Status 404 returned error can't find the container with id df80e094cdc17337426660f6b4a38f19c5a1fc9fe412031a6a80a38a0b30fad4 Nov 03 20:51:15.738422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:15.738318 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" event={"ID":"1081749c-9383-4400-871d-e47f523adb92","Type":"ContainerStarted","Data":"dff5de1b62f8e2e4c52a583a0d5869eaeb5fed0045f01728b5d57b351fb6b3c1"} Nov 03 20:51:15.738422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:15.738412 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" event={"ID":"1081749c-9383-4400-871d-e47f523adb92","Type":"ContainerStarted","Data":"df80e094cdc17337426660f6b4a38f19c5a1fc9fe412031a6a80a38a0b30fad4"} Nov 03 20:51:15.849494 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:15.849433 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" podStartSLOduration=1.849415074 podStartE2EDuration="1.849415074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:15.846800372 +0000 UTC m=+1283.912041857" watchObservedRunningTime="2026-11-03 20:51:15.849415074 +0000 UTC m=+1283.914656548" Nov 03 20:51:16.748578 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:16.748541 5932 generic.go:356] "Generic (PLEG): container finished" podID="1081749c-9383-4400-871d-e47f523adb92" containerID="dff5de1b62f8e2e4c52a583a0d5869eaeb5fed0045f01728b5d57b351fb6b3c1" exitCode=0 Nov 03 20:51:16.749086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:16.748643 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" event={"ID":"1081749c-9383-4400-871d-e47f523adb92","Type":"ContainerDied","Data":"dff5de1b62f8e2e4c52a583a0d5869eaeb5fed0045f01728b5d57b351fb6b3c1"} Nov 03 20:51:17.918093 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:17.918061 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:18.015492 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.015409 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca\") pod \"1081749c-9383-4400-871d-e47f523adb92\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " Nov 03 20:51:18.015492 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.015448 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca\") pod \"1081749c-9383-4400-871d-e47f523adb92\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " Nov 03 20:51:18.015743 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.015618 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc\" (UniqueName: \"kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc\") pod \"1081749c-9383-4400-871d-e47f523adb92\" (UID: \"1081749c-9383-4400-871d-e47f523adb92\") " Nov 03 20:51:18.015971 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.015935 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca" pod "1081749c-9383-4400-871d-e47f523adb92" (UID: "1081749c-9383-4400-871d-e47f523adb92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:18.016180 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.016163 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1081749c-9383-4400-871d-e47f523adb92-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:18.017792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.017769 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca" pod "1081749c-9383-4400-871d-e47f523adb92" (UID: "1081749c-9383-4400-871d-e47f523adb92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:18.017880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.017836 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc" pod "1081749c-9383-4400-871d-e47f523adb92" (UID: "1081749c-9383-4400-871d-e47f523adb92"). InnerVolumeSpecName "kube-api-access-bx8cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:18.117574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.117532 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8cc\" (UniqueName: \"kubernetes.io/projected/1081749c-9383-4400-871d-e47f523adb92-kube-api-access-bx8cc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:18.117574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.117570 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1081749c-9383-4400-871d-e47f523adb92-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:18.765333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.765303 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" Nov 03 20:51:18.765509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.765341 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6" event={"ID":"1081749c-9383-4400-871d-e47f523adb92","Type":"ContainerDied","Data":"df80e094cdc17337426660f6b4a38f19c5a1fc9fe412031a6a80a38a0b30fad4"} Nov 03 20:51:18.765509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:18.765424 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df80e094cdc17337426660f6b4a38f19c5a1fc9fe412031a6a80a38a0b30fad4" Nov 03 20:51:23.046962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:23.046929 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll"] Nov 03 20:51:23.092992 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:23.092956 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad45e676-xf6ll"] Nov 03 20:51:24.038143 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:24.038109 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:24.053256 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:24.053225 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6258d6fc-190f-47a2-a8b8-89060f4fe6c5" path="/var/lib/kubelet/pods/6258d6fc-190f-47a2-a8b8-89060f4fe6c5/volumes" Nov 03 20:51:24.053670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:24.053655 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:24.824305 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:24.824276 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:51:25.625489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.625450 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk"] Nov 03 20:51:25.626264 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.626238 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1081749c-9383-4400-871d-e47f523adb92" containerName="revoke-keys" Nov 03 20:51:25.626366 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.626265 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1081749c-9383-4400-871d-e47f523adb92" containerName="revoke-keys" Nov 03 20:51:25.626482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.626468 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1081749c-9383-4400-871d-e47f523adb92" containerName="revoke-keys" Nov 03 20:51:25.631704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.631681 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.645329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.645302 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk"] Nov 03 20:51:25.649698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.649680 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.788480 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.788444 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk"] Nov 03 20:51:25.820091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820061 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820116 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820153 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820170 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820211 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820227 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820249 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hmm\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820270 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820350 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820384 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820409 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820427 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820458 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhcn\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820474 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820502 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820528 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820548 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.820682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.820586 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.834465 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.834434 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk"] Nov 03 20:51:25.922037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.921954 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.922037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.921988 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922018 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.922302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922046 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.922302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922092 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhcn\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922119 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922156 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922205 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.922585 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922416 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.922585 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922466 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.922585 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922235 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922583 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922592 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922637 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922753 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.922780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922758 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922835 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922868 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922955 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.922982 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923008 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923020 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hmm\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923043 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923053 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923271 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.923530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923417 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.923530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923439 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.923678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.923595 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.925213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.925177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.925304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.925220 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.925413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.925396 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.925484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.925460 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.979831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.979795 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:25.979831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.979818 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.980059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.979882 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhcn\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:25.980059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:25.979890 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hmm\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm\") pod \"e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:26.247250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.247163 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:26.263053 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.263028 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:26.479068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.479038 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp"] Nov 03 20:51:26.484736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.484706 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.532997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.532914 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.532997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.532960 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533032 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533148 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533188 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533258 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533311 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmng\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533359 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.533442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.533428 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.541565 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.541534 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp"] Nov 03 20:51:26.634546 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634511 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.634546 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634550 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634617 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634691 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmng\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634714 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634784 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634819 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634862 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.634884 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.635085 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.635143 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.635173 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.635341 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.635663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.635617 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.637141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.637123 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.637408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.637391 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.691467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.691434 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk"] Nov 03 20:51:26.694847 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.694824 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.695050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.695026 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmng\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.695694 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:26.695664 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd573ec1d_a30a_4188_834c_679db4e61a98.slice/crio-f8172c2fc186b9cfae9ad73c94c2c4101a71c4e02b78855b39f2493a168bc516 WatchSource:0}: Error finding container f8172c2fc186b9cfae9ad73c94c2c4101a71c4e02b78855b39f2493a168bc516: Status 404 returned error can't find the container with id f8172c2fc186b9cfae9ad73c94c2c4101a71c4e02b78855b39f2493a168bc516 Nov 03 20:51:26.698239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.698211 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:26.698317 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.698273 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:26.698317 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.698303 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:26.713314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.713288 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk"] Nov 03 20:51:26.715329 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:26.715295 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e16e9c5_c36e_4026_8e0e_775c6329f41d.slice/crio-1971e3841afbb863bcc41c6c0ed4adbe358a6c1cb9f82798b675cf12f81fb947 WatchSource:0}: Error finding container 1971e3841afbb863bcc41c6c0ed4adbe358a6c1cb9f82798b675cf12f81fb947: Status 404 returned error can't find the container with id 1971e3841afbb863bcc41c6c0ed4adbe358a6c1cb9f82798b675cf12f81fb947 Nov 03 20:51:26.717879 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.717852 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:26.717935 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.717909 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:26.717973 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.717946 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:26.800261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.800215 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:26.852204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.851662 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" event={"ID":"3e16e9c5-c36e-4026-8e0e-775c6329f41d","Type":"ContainerStarted","Data":"56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb"} Nov 03 20:51:26.852204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.851766 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" event={"ID":"3e16e9c5-c36e-4026-8e0e-775c6329f41d","Type":"ContainerStarted","Data":"1971e3841afbb863bcc41c6c0ed4adbe358a6c1cb9f82798b675cf12f81fb947"} Nov 03 20:51:26.854743 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.854705 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" event={"ID":"d573ec1d-a30a-4188-834c-679db4e61a98","Type":"ContainerStarted","Data":"72b9378a0d0994215aa306c4f235090f1bad40340b1532c360edf4cedc09a58a"} Nov 03 20:51:26.854915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.854766 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" event={"ID":"d573ec1d-a30a-4188-834c-679db4e61a98","Type":"ContainerStarted","Data":"f8172c2fc186b9cfae9ad73c94c2c4101a71c4e02b78855b39f2493a168bc516"} Nov 03 20:51:26.966412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:26.966344 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podStartSLOduration=1.966326167 podStartE2EDuration="1.966326167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:26.964046652 +0000 UTC m=+1295.029288126" watchObservedRunningTime="2026-11-03 20:51:26.966326167 +0000 UTC m=+1295.031567641" Nov 03 20:51:27.087020 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.086909 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podStartSLOduration=2.086891249 podStartE2EDuration="2.086891249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:27.081207542 +0000 UTC m=+1295.146449029" watchObservedRunningTime="2026-11-03 20:51:27.086891249 +0000 UTC m=+1295.152132722" Nov 03 20:51:27.247826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.247783 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:27.249485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.249451 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:27.249657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.249518 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:27.269139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.265452 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:27.269139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.265574 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:27.269139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.265617 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:27.269139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.268272 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp"] Nov 03 20:51:27.281987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.281946 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:27.282081 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.282037 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:27.282116 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.282080 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:27.886062 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.886018 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" event={"ID":"ede56218-956c-417d-a362-23f3e37f0f8d","Type":"ContainerStarted","Data":"d81dfab08b900af97b81a1bf0526262460075f2a7e2575451ac4f44979e2a1e3"} Nov 03 20:51:27.886554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:27.886081 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" event={"ID":"ede56218-956c-417d-a362-23f3e37f0f8d","Type":"ContainerStarted","Data":"4145eabe72da68d547ba00fade6e92b02d1e464c6ad0e5b0f83981360494944e"} Nov 03 20:51:28.001970 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.001913 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podStartSLOduration=2.001894346 podStartE2EDuration="2.001894346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:28.000150828 +0000 UTC m=+1296.065392323" watchObservedRunningTime="2026-11-03 20:51:28.001894346 +0000 UTC m=+1296.067135818" Nov 03 20:51:28.250111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.250017 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:28.250396 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.250367 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:28.272057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.272020 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:28.272293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.272268 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:28.801651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.800773 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:28.802784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.802745 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:28.802878 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:28.802812 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:29.076033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.075932 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn"] Nov 03 20:51:29.105429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.105384 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237bfda3-gkwfn"] Nov 03 20:51:29.254882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.254849 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:29.255123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.255095 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:29.267091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.267055 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:29.267360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.267329 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:29.801133 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.801089 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:29.801357 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:29.801148 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:30.101029 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.100939 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b380ef6-c12b-44a5-b3f1-cf621c7d43ed" path="/var/lib/kubelet/pods/6b380ef6-c12b-44a5-b3f1-cf621c7d43ed/volumes" Nov 03 20:51:30.249948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.249913 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:30.250214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.250188 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:30.274078 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.274040 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:30.276756 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.276720 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:30.739799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.739758 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht"] Nov 03 20:51:30.759917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.759887 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.783209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.783180 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.783468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.783449 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpvj\" (UniqueName: \"kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.789823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.789796 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.800754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.800722 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:30.801010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.800983 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:30.808258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.808227 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht"] Nov 03 20:51:30.857783 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.852524 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:30.857783 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.855855 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:51:30.903031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.902975 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.905359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.905333 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpvj\" (UniqueName: \"kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.905902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.905877 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.909185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.909159 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:30.910081 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:30.910056 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:31.010092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.009910 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpvj\" (UniqueName: \"kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj\") pod \"maas-api-revoke-sub-4753bf9b-vtnht\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:31.099450 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.099270 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:31.277712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.264051 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:31.277712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.264115 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:31.277712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.264784 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:31.277712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.264824 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:31.801643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.801591 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:31.801830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.801705 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:31.888666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:31.884454 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht"] Nov 03 20:51:32.015786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.015753 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" event={"ID":"af506dab-2d8d-4dc6-aa76-0f77378601c3","Type":"ContainerStarted","Data":"1b41ea101bb40e371f66a3ab428f34eaa2d5594c0e08cb3b7eb513f300eb2cae"} Nov 03 20:51:32.269649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.268965 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:32.269649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.269030 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:32.273646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.270601 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:32.273646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.270668 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:32.802169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.802130 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:32.804805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:32.804767 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:33.059826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.059780 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" event={"ID":"af506dab-2d8d-4dc6-aa76-0f77378601c3","Type":"ContainerStarted","Data":"c83119d103db057da8df7615e093b8f7738803a14336d5e893cb7f1c8b6a08cc"} Nov 03 20:51:33.169758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.168819 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" podStartSLOduration=3.16880013 podStartE2EDuration="3.16880013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:33.168007 +0000 UTC m=+1301.233248472" watchObservedRunningTime="2026-11-03 20:51:33.16880013 +0000 UTC m=+1301.234041609" Nov 03 20:51:33.252652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.249004 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 20:51:33.252652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.249066 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 20:51:33.263590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.263556 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 20:51:33.263758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.263608 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 20:51:33.801275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.801235 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:33.801459 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:33.801302 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:34.079144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.079037 5932 generic.go:356] "Generic (PLEG): container finished" podID="af506dab-2d8d-4dc6-aa76-0f77378601c3" containerID="c83119d103db057da8df7615e093b8f7738803a14336d5e893cb7f1c8b6a08cc" exitCode=0 Nov 03 20:51:34.079677 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.079142 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" event={"ID":"af506dab-2d8d-4dc6-aa76-0f77378601c3","Type":"ContainerDied","Data":"c83119d103db057da8df7615e093b8f7738803a14336d5e893cb7f1c8b6a08cc"} Nov 03 20:51:34.398476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.398437 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt"] Nov 03 20:51:34.405646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.405602 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.453565 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.453516 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt"] Nov 03 20:51:34.513741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.513054 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:34.535701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.517329 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmh9\" (UniqueName: \"kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.535701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.517482 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.535701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.517535 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.535701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.520101 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:34.535701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.521944 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:51:34.615207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.612600 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:34.615744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.615493 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:34.616558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.616537 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:51:34.620387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.620349 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmh9\" (UniqueName: \"kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.620954 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.620692 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.622015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.621989 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.622116 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.622046 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.625636 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.625607 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.681848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.681772 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmh9\" (UniqueName: \"kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9\") pod \"maas-api-revoke-sub-594ce3d0-mxhxt\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.730980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.730947 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:34.809657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.802312 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 20:51:34.809657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:34.802373 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 20:51:35.134493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.133833 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt"] Nov 03 20:51:35.140737 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:35.140086 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1129f80_eb89_44d4_8f10_f8e302b6f37a.slice/crio-9e0748fad063fa5a1b93122ee5ec76008bdd9000ec4a31ebafbc68df37a5c7a7 WatchSource:0}: Error finding container 9e0748fad063fa5a1b93122ee5ec76008bdd9000ec4a31ebafbc68df37a5c7a7: Status 404 returned error can't find the container with id 9e0748fad063fa5a1b93122ee5ec76008bdd9000ec4a31ebafbc68df37a5c7a7 Nov 03 20:51:35.277196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.277160 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:35.335119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.335086 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca\") pod \"af506dab-2d8d-4dc6-aa76-0f77378601c3\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " Nov 03 20:51:35.335302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.335250 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj\" (UniqueName: \"kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj\") pod \"af506dab-2d8d-4dc6-aa76-0f77378601c3\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " Nov 03 20:51:35.335302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.335279 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca\") pod \"af506dab-2d8d-4dc6-aa76-0f77378601c3\" (UID: \"af506dab-2d8d-4dc6-aa76-0f77378601c3\") " Nov 03 20:51:35.335792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.335767 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca" pod "af506dab-2d8d-4dc6-aa76-0f77378601c3" (UID: "af506dab-2d8d-4dc6-aa76-0f77378601c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:35.337279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.337251 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj" pod "af506dab-2d8d-4dc6-aa76-0f77378601c3" (UID: "af506dab-2d8d-4dc6-aa76-0f77378601c3"). InnerVolumeSpecName "kube-api-access-rkpvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:35.337616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.337601 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca" pod "af506dab-2d8d-4dc6-aa76-0f77378601c3" (UID: "af506dab-2d8d-4dc6-aa76-0f77378601c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:35.437451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.437418 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpvj\" (UniqueName: \"kubernetes.io/projected/af506dab-2d8d-4dc6-aa76-0f77378601c3-kube-api-access-rkpvj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.437451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.437447 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.437682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.437461 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af506dab-2d8d-4dc6-aa76-0f77378601c3-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:35.805560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.805481 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:35.805883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.805860 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:35.806533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:35.806513 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:51:36.097833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:36.097794 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" event={"ID":"d1129f80-eb89-44d4-8f10-f8e302b6f37a","Type":"ContainerStarted","Data":"f2404d0e940bb6a974544be53c53392c8e3bdf217ab3c165779952a3abe2c390"} Nov 03 20:51:36.098000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:36.097860 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" event={"ID":"d1129f80-eb89-44d4-8f10-f8e302b6f37a","Type":"ContainerStarted","Data":"9e0748fad063fa5a1b93122ee5ec76008bdd9000ec4a31ebafbc68df37a5c7a7"} Nov 03 20:51:36.099684 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:36.099659 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" Nov 03 20:51:36.099980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:36.099664 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht" event={"ID":"af506dab-2d8d-4dc6-aa76-0f77378601c3","Type":"ContainerDied","Data":"1b41ea101bb40e371f66a3ab428f34eaa2d5594c0e08cb3b7eb513f300eb2cae"} Nov 03 20:51:36.099980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:36.099797 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b41ea101bb40e371f66a3ab428f34eaa2d5594c0e08cb3b7eb513f300eb2cae" Nov 03 20:51:36.216547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:36.216474 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" podStartSLOduration=2.216460155 podStartE2EDuration="2.216460155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:36.210996527 +0000 UTC m=+1304.276238001" watchObservedRunningTime="2026-11-03 20:51:36.216460155 +0000 UTC m=+1304.281701628" Nov 03 20:51:37.109320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:37.109290 5932 generic.go:356] "Generic (PLEG): container finished" podID="d1129f80-eb89-44d4-8f10-f8e302b6f37a" containerID="f2404d0e940bb6a974544be53c53392c8e3bdf217ab3c165779952a3abe2c390" exitCode=0 Nov 03 20:51:37.109480 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:37.109376 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" event={"ID":"d1129f80-eb89-44d4-8f10-f8e302b6f37a","Type":"ContainerDied","Data":"f2404d0e940bb6a974544be53c53392c8e3bdf217ab3c165779952a3abe2c390"} Nov 03 20:51:38.264676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.264645 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:38.279314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.279286 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca\") pod \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " Nov 03 20:51:38.279439 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.279397 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca\") pod \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " Nov 03 20:51:38.279652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.279617 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9\" (UniqueName: \"kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9\") pod \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\" (UID: \"d1129f80-eb89-44d4-8f10-f8e302b6f37a\") " Nov 03 20:51:38.279784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.279756 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca" pod "d1129f80-eb89-44d4-8f10-f8e302b6f37a" (UID: "d1129f80-eb89-44d4-8f10-f8e302b6f37a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:38.280237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.280210 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:38.281606 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.281582 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca" pod "d1129f80-eb89-44d4-8f10-f8e302b6f37a" (UID: "d1129f80-eb89-44d4-8f10-f8e302b6f37a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:38.281753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.281725 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9" pod "d1129f80-eb89-44d4-8f10-f8e302b6f37a" (UID: "d1129f80-eb89-44d4-8f10-f8e302b6f37a"). InnerVolumeSpecName "kube-api-access-tpmh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:38.381862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.381830 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmh9\" (UniqueName: \"kubernetes.io/projected/d1129f80-eb89-44d4-8f10-f8e302b6f37a-kube-api-access-tpmh9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:38.381862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:38.381862 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1129f80-eb89-44d4-8f10-f8e302b6f37a-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:39.136423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:39.136388 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" Nov 03 20:51:39.136599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:39.136424 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt" event={"ID":"d1129f80-eb89-44d4-8f10-f8e302b6f37a","Type":"ContainerDied","Data":"9e0748fad063fa5a1b93122ee5ec76008bdd9000ec4a31ebafbc68df37a5c7a7"} Nov 03 20:51:39.136599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:39.136505 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e0748fad063fa5a1b93122ee5ec76008bdd9000ec4a31ebafbc68df37a5c7a7" Nov 03 20:51:44.152826 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:51:44.152788 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:51:44.557581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.557494 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm"] Nov 03 20:51:44.558170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.558153 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d1129f80-eb89-44d4-8f10-f8e302b6f37a" containerName="revoke-keys" Nov 03 20:51:44.558209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.558171 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1129f80-eb89-44d4-8f10-f8e302b6f37a" containerName="revoke-keys" Nov 03 20:51:44.558255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.558222 5932 cpu_manager.go:414] "containerMap: removing container" podUID="af506dab-2d8d-4dc6-aa76-0f77378601c3" containerName="revoke-keys" Nov 03 20:51:44.558255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.558227 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af506dab-2d8d-4dc6-aa76-0f77378601c3" containerName="revoke-keys" Nov 03 20:51:44.558328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.558319 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="af506dab-2d8d-4dc6-aa76-0f77378601c3" containerName="revoke-keys" Nov 03 20:51:44.558360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.558335 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1129f80-eb89-44d4-8f10-f8e302b6f37a" containerName="revoke-keys" Nov 03 20:51:44.562129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.562106 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.615685 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.615651 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm"] Nov 03 20:51:44.625535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.625486 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.625719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.625689 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.625932 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.625915 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgc4v\" (UniqueName: \"kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.656637 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.656586 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:51:44.656637 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.656591 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:44.726888 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.726849 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgc4v\" (UniqueName: \"kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.727101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.726914 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.727101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.726973 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.727592 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.727564 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.729587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.729565 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.784533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.784496 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgc4v\" (UniqueName: \"kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v\") pod \"maas-api-revoke-sub-9657716f-srwgm\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:44.876051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:44.876024 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:45.256127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:45.256045 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm"] Nov 03 20:51:45.260543 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:45.260509 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6017bb_c0b7_43cf_b8e7_7678d3f1017c.slice/crio-f95bb2a53811b8cd3c79d6e4efaf7174a47cb37ad4ee0af49ae463f5cd8277af WatchSource:0}: Error finding container f95bb2a53811b8cd3c79d6e4efaf7174a47cb37ad4ee0af49ae463f5cd8277af: Status 404 returned error can't find the container with id f95bb2a53811b8cd3c79d6e4efaf7174a47cb37ad4ee0af49ae463f5cd8277af Nov 03 20:51:46.221964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.221927 5932 generic.go:356] "Generic (PLEG): container finished" podID="ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" containerID="7d2e56d7a4d7433cb6644e4d14bb69acfde65a0fd79709033f7e8a1af2b81328" exitCode=0 Nov 03 20:51:46.222165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.222011 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" event={"ID":"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c","Type":"ContainerDied","Data":"7d2e56d7a4d7433cb6644e4d14bb69acfde65a0fd79709033f7e8a1af2b81328"} Nov 03 20:51:46.222165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.222076 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" event={"ID":"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c","Type":"ContainerStarted","Data":"f95bb2a53811b8cd3c79d6e4efaf7174a47cb37ad4ee0af49ae463f5cd8277af"} Nov 03 20:51:46.768210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.768171 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck"] Nov 03 20:51:46.772431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.772411 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.827515 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.827478 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck"] Nov 03 20:51:46.855617 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.855585 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.855784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.855652 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5bnq\" (UniqueName: \"kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.855873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.855847 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.956703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.956672 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.956911 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.956723 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5bnq\" (UniqueName: \"kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.956911 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.956904 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.957502 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.957476 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:46.959064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:46.959047 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:47.014426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.014388 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5bnq\" (UniqueName: \"kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq\") pod \"maas-api-revoke-sub-00507ce5-dxjck\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:47.052950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.052860 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2"] Nov 03 20:51:47.058531 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.058496 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cde5cf20-7fqj2"] Nov 03 20:51:47.066786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.066755 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg"] Nov 03 20:51:47.070903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.070868 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e60940c-kg9zg"] Nov 03 20:51:47.086114 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.086084 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:47.388865 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.388836 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:47.469490 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.469455 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck"] Nov 03 20:51:47.472834 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:47.472806 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84cb05e_0fb3_4c4b_b9a1_10428ad5d3a4.slice/crio-a32f6c59f1ef418d8ce8447e03bb509b323dd4281dbe75fd28a0b43fcca4f397 WatchSource:0}: Error finding container a32f6c59f1ef418d8ce8447e03bb509b323dd4281dbe75fd28a0b43fcca4f397: Status 404 returned error can't find the container with id a32f6c59f1ef418d8ce8447e03bb509b323dd4281dbe75fd28a0b43fcca4f397 Nov 03 20:51:47.572386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.572351 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca\") pod \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " Nov 03 20:51:47.572554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.572485 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v\" (UniqueName: \"kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v\") pod \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " Nov 03 20:51:47.572657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.572561 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca\") pod \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\" (UID: \"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c\") " Nov 03 20:51:47.572985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.572959 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca" pod "ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" (UID: "ac6017bb-c0b7-43cf-b8e7-7678d3f1017c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:47.575129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.575103 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca" pod "ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" (UID: "ac6017bb-c0b7-43cf-b8e7-7678d3f1017c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:47.575129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.575113 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v" pod "ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" (UID: "ac6017bb-c0b7-43cf-b8e7-7678d3f1017c"). InnerVolumeSpecName "kube-api-access-dgc4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:47.673882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.673847 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.673882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.673875 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgc4v\" (UniqueName: \"kubernetes.io/projected/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-kube-api-access-dgc4v\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:47.673882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:47.673887 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:48.054163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.054129 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc" path="/var/lib/kubelet/pods/d0a2d108-0ca8-4eb0-8d62-bfb29694cdcc/volumes" Nov 03 20:51:48.054639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.054604 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ba66b4-4f18-4c9f-9cd9-99dc282051cb" path="/var/lib/kubelet/pods/d8ba66b4-4f18-4c9f-9cd9-99dc282051cb/volumes" Nov 03 20:51:48.241484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.241386 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" event={"ID":"ac6017bb-c0b7-43cf-b8e7-7678d3f1017c","Type":"ContainerDied","Data":"f95bb2a53811b8cd3c79d6e4efaf7174a47cb37ad4ee0af49ae463f5cd8277af"} Nov 03 20:51:48.241484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.241444 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95bb2a53811b8cd3c79d6e4efaf7174a47cb37ad4ee0af49ae463f5cd8277af" Nov 03 20:51:48.241484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.241399 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm" Nov 03 20:51:48.244157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.243722 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" event={"ID":"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4","Type":"ContainerStarted","Data":"89661fb10a66cd6d0b1b6ae825857fbf7189feb941525ef3bf7a72c3eff6cdd1"} Nov 03 20:51:48.244157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.244106 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" event={"ID":"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4","Type":"ContainerStarted","Data":"a32f6c59f1ef418d8ce8447e03bb509b323dd4281dbe75fd28a0b43fcca4f397"} Nov 03 20:51:48.387919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:48.387865 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" podStartSLOduration=2.3878496350000002 podStartE2EDuration="2.387849635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:48.382429651 +0000 UTC m=+1316.447671126" watchObservedRunningTime="2026-11-03 20:51:48.387849635 +0000 UTC m=+1316.453091108" Nov 03 20:51:49.253974 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:49.253942 5932 generic.go:356] "Generic (PLEG): container finished" podID="a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" containerID="89661fb10a66cd6d0b1b6ae825857fbf7189feb941525ef3bf7a72c3eff6cdd1" exitCode=0 Nov 03 20:51:49.254389 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:49.254023 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" event={"ID":"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4","Type":"ContainerDied","Data":"89661fb10a66cd6d0b1b6ae825857fbf7189feb941525ef3bf7a72c3eff6cdd1"} Nov 03 20:51:50.056803 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.056771 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx"] Nov 03 20:51:50.094174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.094135 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f2f6d61-pp4wx"] Nov 03 20:51:50.421894 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.421862 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:50.512860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.512826 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca\") pod \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " Nov 03 20:51:50.513031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.513007 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq\" (UniqueName: \"kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq\") pod \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " Nov 03 20:51:50.513144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.513131 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca\") pod \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\" (UID: \"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4\") " Nov 03 20:51:50.513464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.513430 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca" pod "a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" (UID: "a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:51:50.515230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.515204 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq" pod "a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" (UID: "a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4"). InnerVolumeSpecName "kube-api-access-m5bnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:51:50.515419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.515396 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca" pod "a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" (UID: "a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:51:50.615478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.615380 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:50.615478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.615420 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:50.615478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:50.615430 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5bnq\" (UniqueName: \"kubernetes.io/projected/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4-kube-api-access-m5bnq\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:51:51.271222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:51.271187 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" Nov 03 20:51:51.271222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:51.271217 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck" event={"ID":"a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4","Type":"ContainerDied","Data":"a32f6c59f1ef418d8ce8447e03bb509b323dd4281dbe75fd28a0b43fcca4f397"} Nov 03 20:51:51.271451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:51.271269 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32f6c59f1ef418d8ce8447e03bb509b323dd4281dbe75fd28a0b43fcca4f397" Nov 03 20:51:52.053740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:52.053705 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca762bf-f901-433d-9778-8f9ab063db33" path="/var/lib/kubelet/pods/9ca762bf-f901-433d-9778-8f9ab063db33/volumes" Nov 03 20:51:52.276744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:52.276714 5932 scope.go:122] "RemoveContainer" containerID="c51614609c0158a59f83b62b36664afcfacc1afc541ef0581e56da1ce7d43ab9" Nov 03 20:51:52.289527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:52.289506 5932 scope.go:122] "RemoveContainer" containerID="09c0d2177a88b35a4cf4654ff12dda9d410a7c5b51944b3cebc7c78a65544ca9" Nov 03 20:51:52.301693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:52.301675 5932 scope.go:122] "RemoveContainer" containerID="5481827a24e3fb15a7386326168d30b42bb7c98e1c8832218df69d91da8059e5" Nov 03 20:51:57.338516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.338479 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684"] Nov 03 20:51:57.339282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.339264 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" containerName="revoke-keys" Nov 03 20:51:57.339329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.339285 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" containerName="revoke-keys" Nov 03 20:51:57.339329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.339301 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" containerName="revoke-keys" Nov 03 20:51:57.339329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.339307 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" containerName="revoke-keys" Nov 03 20:51:57.339436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.339427 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" containerName="revoke-keys" Nov 03 20:51:57.339476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.339448 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" containerName="revoke-keys" Nov 03 20:51:57.343353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.343334 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.393054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.393019 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:51:57.393276 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.393255 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:51:57.395712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.395686 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684"] Nov 03 20:51:57.507004 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.506954 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.507282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.507022 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.507282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.507080 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qvr\" (UniqueName: \"kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.609018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.608916 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.609018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.608966 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.609018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.609002 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qvr\" (UniqueName: \"kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.609707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.609687 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.611400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.611375 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.667912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.667881 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qvr\" (UniqueName: \"kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr\") pod \"maas-api-revoke-sub-55d26abf-55684\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.762729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.762692 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj"] Nov 03 20:51:57.768304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.768281 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.810589 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.810549 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.810802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.810606 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.810802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.810758 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.810802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.810792 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.810950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.810820 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.810985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.810947 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kcpd\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.811056 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.811040 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.811094 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.811072 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.811139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.811098 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.827491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.827440 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj"] Nov 03 20:51:57.912102 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912056 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912102 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912108 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912146 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912161 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912176 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912215 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kcpd\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912342 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912379 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912414 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912531 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912559 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912608 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.912876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.912718 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.913339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.913308 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.914860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.914836 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.914972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.914947 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.958281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.958248 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:51:57.970040 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.970012 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:57.970290 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:57.970271 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kcpd\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:58.083139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:58.083104 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:51:58.337194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:58.337114 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684"] Nov 03 20:51:58.344063 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:58.344019 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5baad81_1a62_41d3_a967_be26eae51b29.slice/crio-44267e91a55936af0df3d5409fca67d273266d43486b517f2157f70bbd5abe8b WatchSource:0}: Error finding container 44267e91a55936af0df3d5409fca67d273266d43486b517f2157f70bbd5abe8b: Status 404 returned error can't find the container with id 44267e91a55936af0df3d5409fca67d273266d43486b517f2157f70bbd5abe8b Nov 03 20:51:58.475770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:58.475738 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj"] Nov 03 20:51:58.479286 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:51:58.479256 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82786355_cc14_4d9f_9912_e1c13d7e3b62.slice/crio-e981826c56f48c9359db48877b94b8a2406e2660acab5c57400e2da52269d8a1 WatchSource:0}: Error finding container e981826c56f48c9359db48877b94b8a2406e2660acab5c57400e2da52269d8a1: Status 404 returned error can't find the container with id e981826c56f48c9359db48877b94b8a2406e2660acab5c57400e2da52269d8a1 Nov 03 20:51:58.482026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:58.481992 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:58.482095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:58.482058 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:58.482095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:58.482089 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:51:59.359003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:59.358966 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" event={"ID":"82786355-cc14-4d9f-9912-e1c13d7e3b62","Type":"ContainerStarted","Data":"86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e"} Nov 03 20:51:59.359427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:59.359032 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" event={"ID":"82786355-cc14-4d9f-9912-e1c13d7e3b62","Type":"ContainerStarted","Data":"e981826c56f48c9359db48877b94b8a2406e2660acab5c57400e2da52269d8a1"} Nov 03 20:51:59.361012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:59.360986 5932 generic.go:356] "Generic (PLEG): container finished" podID="a5baad81-1a62-41d3-a967-be26eae51b29" containerID="f7dc477aa2c1857e4b303f7bf46c2dac2627c5fb8e2e02d5863b5e9ac8d0850d" exitCode=0 Nov 03 20:51:59.361134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:59.361042 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" event={"ID":"a5baad81-1a62-41d3-a967-be26eae51b29","Type":"ContainerDied","Data":"f7dc477aa2c1857e4b303f7bf46c2dac2627c5fb8e2e02d5863b5e9ac8d0850d"} Nov 03 20:51:59.361134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:59.361073 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" event={"ID":"a5baad81-1a62-41d3-a967-be26eae51b29","Type":"ContainerStarted","Data":"44267e91a55936af0df3d5409fca67d273266d43486b517f2157f70bbd5abe8b"} Nov 03 20:51:59.519577 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:51:59.519520 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podStartSLOduration=2.519505221 podStartE2EDuration="2.519505221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:59.517738844 +0000 UTC m=+1327.582980323" watchObservedRunningTime="2026-11-03 20:51:59.519505221 +0000 UTC m=+1327.584746693" Nov 03 20:52:00.084207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.084173 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:52:00.085885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.085859 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:52:00.086014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.085911 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:52:00.564593 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.564568 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:52:00.670268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.670239 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca\") pod \"a5baad81-1a62-41d3-a967-be26eae51b29\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " Nov 03 20:52:00.670479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.670283 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr\" (UniqueName: \"kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr\") pod \"a5baad81-1a62-41d3-a967-be26eae51b29\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " Nov 03 20:52:00.670479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.670424 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca\") pod \"a5baad81-1a62-41d3-a967-be26eae51b29\" (UID: \"a5baad81-1a62-41d3-a967-be26eae51b29\") " Nov 03 20:52:00.670610 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.670591 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca" pod "a5baad81-1a62-41d3-a967-be26eae51b29" (UID: "a5baad81-1a62-41d3-a967-be26eae51b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:00.670953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.670935 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:00.672545 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.672520 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr" pod "a5baad81-1a62-41d3-a967-be26eae51b29" (UID: "a5baad81-1a62-41d3-a967-be26eae51b29"). InnerVolumeSpecName "kube-api-access-t5qvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:00.672545 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.672526 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca" pod "a5baad81-1a62-41d3-a967-be26eae51b29" (UID: "a5baad81-1a62-41d3-a967-be26eae51b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:00.772170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.772133 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5baad81-1a62-41d3-a967-be26eae51b29-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:00.772170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:00.772166 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5qvr\" (UniqueName: \"kubernetes.io/projected/a5baad81-1a62-41d3-a967-be26eae51b29-kube-api-access-t5qvr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:01.052058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.051970 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j"] Nov 03 20:52:01.063947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.060761 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e01f4d1-fcw8j"] Nov 03 20:52:01.083935 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.083898 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:52:01.084087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.083973 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:52:01.379387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.379361 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" Nov 03 20:52:01.379581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.379390 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684" event={"ID":"a5baad81-1a62-41d3-a967-be26eae51b29","Type":"ContainerDied","Data":"44267e91a55936af0df3d5409fca67d273266d43486b517f2157f70bbd5abe8b"} Nov 03 20:52:01.379581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:01.379469 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44267e91a55936af0df3d5409fca67d273266d43486b517f2157f70bbd5abe8b" Nov 03 20:52:02.056925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.056895 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4985f4-6814-463d-82f9-31b738108909" path="/var/lib/kubelet/pods/9c4985f4-6814-463d-82f9-31b738108909/volumes" Nov 03 20:52:02.084329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.084288 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:52:02.084670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.084353 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:52:02.473668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.473637 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk"] Nov 03 20:52:02.474317 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.474300 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a5baad81-1a62-41d3-a967-be26eae51b29" containerName="revoke-keys" Nov 03 20:52:02.474364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.474319 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5baad81-1a62-41d3-a967-be26eae51b29" containerName="revoke-keys" Nov 03 20:52:02.474487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.474478 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5baad81-1a62-41d3-a967-be26eae51b29" containerName="revoke-keys" Nov 03 20:52:02.478476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.478458 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.496441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.496413 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.496599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.496571 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.496706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.496691 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n668p\" (UniqueName: \"kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.554017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.553975 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk"] Nov 03 20:52:02.575143 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.575110 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:02.575143 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.575122 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:52:02.598040 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.598006 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n668p\" (UniqueName: \"kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.598202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.598162 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.598292 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.598275 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.598849 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.598826 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.600679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.600654 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.656261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.656228 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n668p\" (UniqueName: \"kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p\") pod \"maas-api-revoke-sub-2a4ebd53-hpfpk\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:02.792230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:02.792108 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:03.051607 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.051512 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw"] Nov 03 20:52:03.057420 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.057391 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh"] Nov 03 20:52:03.061280 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.061256 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d39ed54-f9trw"] Nov 03 20:52:03.084479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.084449 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:52:03.084654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.084498 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:52:03.097122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.097093 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9aef069b-99slh"] Nov 03 20:52:03.200844 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.200801 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk"] Nov 03 20:52:03.205043 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:03.205014 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a101eb8_9bd7_4410_9500_91db0893592c.slice/crio-9fc540c99069ddfc1313c815f204acd9a4901c8a44e6d11d1727e7141981d15c WatchSource:0}: Error finding container 9fc540c99069ddfc1313c815f204acd9a4901c8a44e6d11d1727e7141981d15c: Status 404 returned error can't find the container with id 9fc540c99069ddfc1313c815f204acd9a4901c8a44e6d11d1727e7141981d15c Nov 03 20:52:03.397369 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:03.397337 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" event={"ID":"2a101eb8-9bd7-4410-9500-91db0893592c","Type":"ContainerStarted","Data":"9fc540c99069ddfc1313c815f204acd9a4901c8a44e6d11d1727e7141981d15c"} Nov 03 20:52:04.057416 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.057336 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ec0d12-02db-4f39-8cd8-fb84dae4cb94" path="/var/lib/kubelet/pods/33ec0d12-02db-4f39-8cd8-fb84dae4cb94/volumes" Nov 03 20:52:04.057956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.057854 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84dca63f-55ab-4e10-b46b-cd34fd1410ed" path="/var/lib/kubelet/pods/84dca63f-55ab-4e10-b46b-cd34fd1410ed/volumes" Nov 03 20:52:04.084289 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.084256 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:52:04.084429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.084304 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:52:04.412613 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.412577 5932 generic.go:356] "Generic (PLEG): container finished" podID="2a101eb8-9bd7-4410-9500-91db0893592c" containerID="aa186e472e517ecba7748dab9a6e6887f726aecf2d4e114a66555220738fe446" exitCode=0 Nov 03 20:52:04.412834 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.412657 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" event={"ID":"2a101eb8-9bd7-4410-9500-91db0893592c","Type":"ContainerDied","Data":"aa186e472e517ecba7748dab9a6e6887f726aecf2d4e114a66555220738fe446"} Nov 03 20:52:04.563378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.563342 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr"] Nov 03 20:52:04.567748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.567731 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.623466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.623422 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.623684 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.623527 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s75d\" (UniqueName: \"kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.623684 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.623579 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.639549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.639432 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr"] Nov 03 20:52:04.725947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.725850 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.725947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.725930 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s75d\" (UniqueName: \"kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.726156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.725997 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.726760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.726727 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.728413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.728387 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.783820 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.783790 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s75d\" (UniqueName: \"kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d\") pod \"maas-api-revoke-sub-18e6420a-9jmmr\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:04.882039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:04.882003 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:05.084461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.084371 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 20:52:05.084461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.084430 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 20:52:05.261444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.261409 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr"] Nov 03 20:52:05.265424 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:05.265391 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e82ac3_ba3c_4789_a24f_c413746c0d14.slice/crio-044ef24ad6b5b74e9623151fba0935399c32e01d5ac6b15f120884fe1e63a77e WatchSource:0}: Error finding container 044ef24ad6b5b74e9623151fba0935399c32e01d5ac6b15f120884fe1e63a77e: Status 404 returned error can't find the container with id 044ef24ad6b5b74e9623151fba0935399c32e01d5ac6b15f120884fe1e63a77e Nov 03 20:52:05.423181 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.423146 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" event={"ID":"99e82ac3-ba3c-4789-a24f-c413746c0d14","Type":"ContainerStarted","Data":"044ef24ad6b5b74e9623151fba0935399c32e01d5ac6b15f120884fe1e63a77e"} Nov 03 20:52:05.574259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.574231 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:05.639753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.639720 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p\" (UniqueName: \"kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p\") pod \"2a101eb8-9bd7-4410-9500-91db0893592c\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " Nov 03 20:52:05.639955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.639761 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca\") pod \"2a101eb8-9bd7-4410-9500-91db0893592c\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " Nov 03 20:52:05.639955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.639869 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca\") pod \"2a101eb8-9bd7-4410-9500-91db0893592c\" (UID: \"2a101eb8-9bd7-4410-9500-91db0893592c\") " Nov 03 20:52:05.640227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.640199 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca" pod "2a101eb8-9bd7-4410-9500-91db0893592c" (UID: "2a101eb8-9bd7-4410-9500-91db0893592c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:05.640639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.640605 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:05.642021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.641986 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca" pod "2a101eb8-9bd7-4410-9500-91db0893592c" (UID: "2a101eb8-9bd7-4410-9500-91db0893592c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:05.642125 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.642065 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p" pod "2a101eb8-9bd7-4410-9500-91db0893592c" (UID: "2a101eb8-9bd7-4410-9500-91db0893592c"). InnerVolumeSpecName "kube-api-access-n668p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:05.742630 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.742586 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n668p\" (UniqueName: \"kubernetes.io/projected/2a101eb8-9bd7-4410-9500-91db0893592c-kube-api-access-n668p\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:05.742739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:05.742642 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a101eb8-9bd7-4410-9500-91db0893592c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:06.088082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.088051 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:52:06.088544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.088451 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:52:06.089099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.089078 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 20:52:06.442793 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.442762 5932 generic.go:356] "Generic (PLEG): container finished" podID="99e82ac3-ba3c-4789-a24f-c413746c0d14" containerID="fe2cc2d8f16886f92f31744596ad0c501029b96a44c3a756c16c3de3196fd928" exitCode=0 Nov 03 20:52:06.442955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.442847 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" event={"ID":"99e82ac3-ba3c-4789-a24f-c413746c0d14","Type":"ContainerDied","Data":"fe2cc2d8f16886f92f31744596ad0c501029b96a44c3a756c16c3de3196fd928"} Nov 03 20:52:06.444331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.444315 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" Nov 03 20:52:06.444427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.444350 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk" event={"ID":"2a101eb8-9bd7-4410-9500-91db0893592c","Type":"ContainerDied","Data":"9fc540c99069ddfc1313c815f204acd9a4901c8a44e6d11d1727e7141981d15c"} Nov 03 20:52:06.444427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:06.444396 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc540c99069ddfc1313c815f204acd9a4901c8a44e6d11d1727e7141981d15c" Nov 03 20:52:07.623412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.623387 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:07.681361 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.681334 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca\") pod \"99e82ac3-ba3c-4789-a24f-c413746c0d14\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " Nov 03 20:52:07.681544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.681387 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d\" (UniqueName: \"kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d\") pod \"99e82ac3-ba3c-4789-a24f-c413746c0d14\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " Nov 03 20:52:07.681668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.681600 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca\") pod \"99e82ac3-ba3c-4789-a24f-c413746c0d14\" (UID: \"99e82ac3-ba3c-4789-a24f-c413746c0d14\") " Nov 03 20:52:07.681744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.681722 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca" pod "99e82ac3-ba3c-4789-a24f-c413746c0d14" (UID: "99e82ac3-ba3c-4789-a24f-c413746c0d14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:07.682193 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.682161 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:07.683540 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.683521 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d" pod "99e82ac3-ba3c-4789-a24f-c413746c0d14" (UID: "99e82ac3-ba3c-4789-a24f-c413746c0d14"). InnerVolumeSpecName "kube-api-access-8s75d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:07.683687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.683666 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca" pod "99e82ac3-ba3c-4789-a24f-c413746c0d14" (UID: "99e82ac3-ba3c-4789-a24f-c413746c0d14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:07.783751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.783666 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e82ac3-ba3c-4789-a24f-c413746c0d14-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:07.783751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:07.783697 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s75d\" (UniqueName: \"kubernetes.io/projected/99e82ac3-ba3c-4789-a24f-c413746c0d14-kube-api-access-8s75d\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:08.462987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:08.462958 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" Nov 03 20:52:08.463156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:08.462992 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr" event={"ID":"99e82ac3-ba3c-4789-a24f-c413746c0d14","Type":"ContainerDied","Data":"044ef24ad6b5b74e9623151fba0935399c32e01d5ac6b15f120884fe1e63a77e"} Nov 03 20:52:08.463156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:08.463044 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044ef24ad6b5b74e9623151fba0935399c32e01d5ac6b15f120884fe1e63a77e" Nov 03 20:52:10.053049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:10.053018 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj"] Nov 03 20:52:10.095075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:10.095027 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b12aa87-sgwdj"] Nov 03 20:52:11.513062 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513031 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l"] Nov 03 20:52:11.513702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513685 5932 cpu_manager.go:414] "containerMap: removing container" podUID="99e82ac3-ba3c-4789-a24f-c413746c0d14" containerName="revoke-keys" Nov 03 20:52:11.513702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513702 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e82ac3-ba3c-4789-a24f-c413746c0d14" containerName="revoke-keys" Nov 03 20:52:11.513847 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513732 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2a101eb8-9bd7-4410-9500-91db0893592c" containerName="revoke-keys" Nov 03 20:52:11.513847 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513738 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a101eb8-9bd7-4410-9500-91db0893592c" containerName="revoke-keys" Nov 03 20:52:11.513847 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513836 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a101eb8-9bd7-4410-9500-91db0893592c" containerName="revoke-keys" Nov 03 20:52:11.513998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.513853 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e82ac3-ba3c-4789-a24f-c413746c0d14" containerName="revoke-keys" Nov 03 20:52:11.517976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.517950 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.621389 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.621346 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l"] Nov 03 20:52:11.679190 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.679123 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.679533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.679509 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.679696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.679679 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.679900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.679863 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.685584 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.685507 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.727681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.685928 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.766769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.766657 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwdkj\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.767021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.766992 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.767176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.767158 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869003 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869245 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869058 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869245 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869107 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwdkj\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869245 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869138 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869245 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869171 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869490 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869259 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869490 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869326 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869490 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869352 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869669 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869673 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869442 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869791 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.869962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.869932 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.870119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.870092 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.871798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.871766 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.872085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.872063 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.926833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.926799 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:11.927010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:11.926847 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwdkj\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj\") pod \"e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:12.053415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.053332 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149d7d57-4fdb-47e2-a29d-33a7c10e848f" path="/var/lib/kubelet/pods/149d7d57-4fdb-47e2-a29d-33a7c10e848f/volumes" Nov 03 20:52:12.148978 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.148937 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:12.341869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.341837 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj"] Nov 03 20:52:12.348842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.348736 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.365706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.365673 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj"] Nov 03 20:52:12.420428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.413128 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:12.420428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.413281 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:52:12.503648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.492109 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.503648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.492158 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzt6\" (UniqueName: \"kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.503648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.492210 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.607714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.603303 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.607714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.603356 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzt6\" (UniqueName: \"kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.607714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.603415 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.607714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.604531 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.630636 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.630583 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.672599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.672526 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzt6\" (UniqueName: \"kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6\") pod \"maas-api-revoke-sub-a6b15f3f-hhqfj\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:12.839686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:12.836175 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:13.441913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:13.435210 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:13.441913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:13.435308 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:13.441913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:13.435353 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:13.468706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:13.450985 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l"] Nov 03 20:52:13.825647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:13.822210 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" event={"ID":"96e07592-beae-4eb9-a59b-500ed6d90744","Type":"ContainerStarted","Data":"e9f8029de220b108a09da105e6c994774d8931e04a12c539d1829215fb3b50f6"} Nov 03 20:52:14.174608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:14.174087 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:14.269282 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:52:14.269200 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:52:14.282848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:14.282809 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:14.557059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:14.557023 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj"] Nov 03 20:52:14.561912 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:14.561882 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc015ab1b_0a13_4ca9_85c4_b938cc71cd93.slice/crio-92af5f406428655f4c3ce640dc95274c326fec0744ba4b49d64bb8b6007f4197 WatchSource:0}: Error finding container 92af5f406428655f4c3ce640dc95274c326fec0744ba4b49d64bb8b6007f4197: Status 404 returned error can't find the container with id 92af5f406428655f4c3ce640dc95274c326fec0744ba4b49d64bb8b6007f4197 Nov 03 20:52:14.852739 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:14.852652 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" event={"ID":"c015ab1b-0a13-4ca9-85c4-b938cc71cd93","Type":"ContainerStarted","Data":"92af5f406428655f4c3ce640dc95274c326fec0744ba4b49d64bb8b6007f4197"} Nov 03 20:52:14.854782 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:14.854753 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" event={"ID":"96e07592-beae-4eb9-a59b-500ed6d90744","Type":"ContainerStarted","Data":"b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8"} Nov 03 20:52:14.972948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:14.972895 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podStartSLOduration=3.972881749 podStartE2EDuration="3.972881749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:14.967879674 +0000 UTC m=+1343.033121147" watchObservedRunningTime="2026-11-03 20:52:14.972881749 +0000 UTC m=+1343.038123222" Nov 03 20:52:15.151024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:15.150991 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:15.152560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:15.152529 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:52:15.152718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:15.152586 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:52:15.865719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:15.865600 5932 generic.go:356] "Generic (PLEG): container finished" podID="c015ab1b-0a13-4ca9-85c4-b938cc71cd93" containerID="7b30892aef707dfd43994f1fa1ee39f7eb2c57b561b76f14afa0b9df676f80c8" exitCode=0 Nov 03 20:52:15.865719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:15.865678 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" event={"ID":"c015ab1b-0a13-4ca9-85c4-b938cc71cd93","Type":"ContainerDied","Data":"7b30892aef707dfd43994f1fa1ee39f7eb2c57b561b76f14afa0b9df676f80c8"} Nov 03 20:52:16.151176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:16.151129 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:52:16.151379 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:16.151203 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:52:17.057941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.057910 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5"] Nov 03 20:52:17.063441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.063414 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33d5d111-5cds5"] Nov 03 20:52:17.092462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.092435 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:17.151169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.151084 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:52:17.151169 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.151135 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:52:17.197989 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.197954 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca\") pod \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " Nov 03 20:52:17.198154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.198008 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca\") pod \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " Nov 03 20:52:17.198234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.198223 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6\" (UniqueName: \"kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6\") pod \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\" (UID: \"c015ab1b-0a13-4ca9-85c4-b938cc71cd93\") " Nov 03 20:52:17.198388 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.198359 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca" pod "c015ab1b-0a13-4ca9-85c4-b938cc71cd93" (UID: "c015ab1b-0a13-4ca9-85c4-b938cc71cd93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:17.198700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.198670 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:17.200161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.200141 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca" pod "c015ab1b-0a13-4ca9-85c4-b938cc71cd93" (UID: "c015ab1b-0a13-4ca9-85c4-b938cc71cd93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:17.200244 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.200210 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6" pod "c015ab1b-0a13-4ca9-85c4-b938cc71cd93" (UID: "c015ab1b-0a13-4ca9-85c4-b938cc71cd93"). InnerVolumeSpecName "kube-api-access-xlzt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:17.299784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.299752 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzt6\" (UniqueName: \"kubernetes.io/projected/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-kube-api-access-xlzt6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:17.299784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.299780 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c015ab1b-0a13-4ca9-85c4-b938cc71cd93-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:17.566705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.566611 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr"] Nov 03 20:52:17.567325 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.567310 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c015ab1b-0a13-4ca9-85c4-b938cc71cd93" containerName="revoke-keys" Nov 03 20:52:17.567385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.567326 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c015ab1b-0a13-4ca9-85c4-b938cc71cd93" containerName="revoke-keys" Nov 03 20:52:17.567430 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.567421 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c015ab1b-0a13-4ca9-85c4-b938cc71cd93" containerName="revoke-keys" Nov 03 20:52:17.571329 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.571307 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.651650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.648941 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr"] Nov 03 20:52:17.704878 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.704835 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.705053 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.704908 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.705136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.705101 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kccmp\" (UniqueName: \"kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.756193 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.756157 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z"] Nov 03 20:52:17.760925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.760907 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.807000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.806946 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.807174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.807087 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.807174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.807117 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.807284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.807258 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.807352 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.807319 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsfj2\" (UniqueName: \"kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.807431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.807413 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kccmp\" (UniqueName: \"kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.807800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.807781 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.809431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.809413 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.850709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.845774 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z"] Nov 03 20:52:17.865493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.865460 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kccmp\" (UniqueName: \"kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp\") pod \"maas-api-revoke-sub-60365f29-r5fpr\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.885032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.885005 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" Nov 03 20:52:17.885189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.885032 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj" event={"ID":"c015ab1b-0a13-4ca9-85c4-b938cc71cd93","Type":"ContainerDied","Data":"92af5f406428655f4c3ce640dc95274c326fec0744ba4b49d64bb8b6007f4197"} Nov 03 20:52:17.885189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.885081 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92af5f406428655f4c3ce640dc95274c326fec0744ba4b49d64bb8b6007f4197" Nov 03 20:52:17.886703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.886685 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:17.908899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.908382 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.908899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.908491 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.908899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.908533 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsfj2\" (UniqueName: \"kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.909459 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.909431 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.910769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.910750 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:17.967566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:17.967104 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsfj2\" (UniqueName: \"kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2\") pod \"maas-api-revoke-sub-f66e21fa-nzl6z\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:18.061668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.061190 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="594275a5-5aa9-4cd8-8f96-dd5ff5e7a197" path="/var/lib/kubelet/pods/594275a5-5aa9-4cd8-8f96-dd5ff5e7a197/volumes" Nov 03 20:52:18.075095 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.075069 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:18.151432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.151047 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:52:18.151432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.151104 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:52:18.293866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.293835 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr"] Nov 03 20:52:18.478210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.478120 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z"] Nov 03 20:52:18.894984 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.894946 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" event={"ID":"b9176006-ca95-475c-b441-04e91b3a8861","Type":"ContainerStarted","Data":"3e9070109785ac1b648337619c0798eafd1e16410afcb62958dbad700db0010a"} Nov 03 20:52:18.896722 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.896697 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" event={"ID":"c00eeb35-be75-470f-82ce-85c54a7b117d","Type":"ContainerStarted","Data":"19a1c443b44cbf2dc1818c35fee1947e9d740e48362c63d8e658510235e9f92f"} Nov 03 20:52:18.896872 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:18.896736 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" event={"ID":"c00eeb35-be75-470f-82ce-85c54a7b117d","Type":"ContainerStarted","Data":"b0e17fa85bc696e8a339c2d896588bf4dff208660fcecd9a5869744d4f5615d8"} Nov 03 20:52:19.006916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.006868 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" podStartSLOduration=2.006854574 podStartE2EDuration="2.006854574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:19.005997974 +0000 UTC m=+1347.071239447" watchObservedRunningTime="2026-11-03 20:52:19.006854574 +0000 UTC m=+1347.072096047" Nov 03 20:52:19.058064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.058037 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w"] Nov 03 20:52:19.065110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.065083 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3fe72be6-7hr6w"] Nov 03 20:52:19.151572 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.151536 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:52:19.151832 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.151596 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:52:19.152962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.152937 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl"] Nov 03 20:52:19.157414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.157395 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.201602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.201552 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl"] Nov 03 20:52:19.258226 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.257798 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.258226 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.258012 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.258492 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.258240 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wdnd\" (UniqueName: \"kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.359065 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.359024 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdnd\" (UniqueName: \"kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.359281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.359111 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.359281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.359206 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.359897 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.359875 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.361681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.361652 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.416684 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.416574 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wdnd\" (UniqueName: \"kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd\") pod \"maas-api-revoke-sub-0acd8581-r6vtl\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.473294 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.473258 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:19.889786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.889516 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl"] Nov 03 20:52:19.895735 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:19.895707 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffdd307_0c75_4868_8f5f_f725b9b39c00.slice/crio-2dafe91b3d60931422c96cd7e5145e4e881c8d2f9c3c74928ad935c57174a70e WatchSource:0}: Error finding container 2dafe91b3d60931422c96cd7e5145e4e881c8d2f9c3c74928ad935c57174a70e: Status 404 returned error can't find the container with id 2dafe91b3d60931422c96cd7e5145e4e881c8d2f9c3c74928ad935c57174a70e Nov 03 20:52:19.917230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.917200 5932 generic.go:356] "Generic (PLEG): container finished" podID="c00eeb35-be75-470f-82ce-85c54a7b117d" containerID="19a1c443b44cbf2dc1818c35fee1947e9d740e48362c63d8e658510235e9f92f" exitCode=0 Nov 03 20:52:19.917360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.917341 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" event={"ID":"c00eeb35-be75-470f-82ce-85c54a7b117d","Type":"ContainerDied","Data":"19a1c443b44cbf2dc1818c35fee1947e9d740e48362c63d8e658510235e9f92f"} Nov 03 20:52:19.919391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.919365 5932 generic.go:356] "Generic (PLEG): container finished" podID="b9176006-ca95-475c-b441-04e91b3a8861" containerID="d862c1d79f7e741b0a5e9b41f19ff8ab8c21f8694e156ac906a5d9252f563f2a" exitCode=0 Nov 03 20:52:19.919558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.919449 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" event={"ID":"b9176006-ca95-475c-b441-04e91b3a8861","Type":"ContainerDied","Data":"d862c1d79f7e741b0a5e9b41f19ff8ab8c21f8694e156ac906a5d9252f563f2a"} Nov 03 20:52:19.920526 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:19.920506 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" event={"ID":"3ffdd307-0c75-4868-8f5f-f725b9b39c00","Type":"ContainerStarted","Data":"2dafe91b3d60931422c96cd7e5145e4e881c8d2f9c3c74928ad935c57174a70e"} Nov 03 20:52:20.057025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.056988 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff087102-1bb8-41ce-ae9e-09d2477d8560" path="/var/lib/kubelet/pods/ff087102-1bb8-41ce-ae9e-09d2477d8560/volumes" Nov 03 20:52:20.057376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.057359 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj"] Nov 03 20:52:20.083249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.083158 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308ecf28-fxnwj"] Nov 03 20:52:20.153605 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.153438 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 20:52:20.153605 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.153500 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 20:52:20.930028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.929987 5932 generic.go:356] "Generic (PLEG): container finished" podID="3ffdd307-0c75-4868-8f5f-f725b9b39c00" containerID="c7a32f086d3f29d73ded5733a32055e7064fe0bdb61593f53e13da051093e823" exitCode=0 Nov 03 20:52:20.930177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:20.930066 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" event={"ID":"3ffdd307-0c75-4868-8f5f-f725b9b39c00","Type":"ContainerDied","Data":"c7a32f086d3f29d73ded5733a32055e7064fe0bdb61593f53e13da051093e823"} Nov 03 20:52:21.194705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.194674 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:21.277551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.272608 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca\") pod \"c00eeb35-be75-470f-82ce-85c54a7b117d\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " Nov 03 20:52:21.277551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.273113 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp\" (UniqueName: \"kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp\") pod \"c00eeb35-be75-470f-82ce-85c54a7b117d\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " Nov 03 20:52:21.277551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.277431 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca" pod "c00eeb35-be75-470f-82ce-85c54a7b117d" (UID: "c00eeb35-be75-470f-82ce-85c54a7b117d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:21.278157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.278098 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca\") pod \"c00eeb35-be75-470f-82ce-85c54a7b117d\" (UID: \"c00eeb35-be75-470f-82ce-85c54a7b117d\") " Nov 03 20:52:21.280736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.278872 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:21.285065 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.285033 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp" pod "c00eeb35-be75-470f-82ce-85c54a7b117d" (UID: "c00eeb35-be75-470f-82ce-85c54a7b117d"). InnerVolumeSpecName "kube-api-access-kccmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:21.285842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.285822 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca" pod "c00eeb35-be75-470f-82ce-85c54a7b117d" (UID: "c00eeb35-be75-470f-82ce-85c54a7b117d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:21.287148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.287130 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:21.379753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.379717 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca\") pod \"b9176006-ca95-475c-b441-04e91b3a8861\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " Nov 03 20:52:21.379935 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.379790 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca\") pod \"b9176006-ca95-475c-b441-04e91b3a8861\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " Nov 03 20:52:21.379935 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.379819 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2\" (UniqueName: \"kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2\") pod \"b9176006-ca95-475c-b441-04e91b3a8861\" (UID: \"b9176006-ca95-475c-b441-04e91b3a8861\") " Nov 03 20:52:21.380391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.380362 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca" pod "b9176006-ca95-475c-b441-04e91b3a8861" (UID: "b9176006-ca95-475c-b441-04e91b3a8861"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:21.380687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.380618 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:21.380764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.380695 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c00eeb35-be75-470f-82ce-85c54a7b117d-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:21.380764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.380712 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kccmp\" (UniqueName: \"kubernetes.io/projected/c00eeb35-be75-470f-82ce-85c54a7b117d-kube-api-access-kccmp\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:21.381990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.381961 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca" pod "b9176006-ca95-475c-b441-04e91b3a8861" (UID: "b9176006-ca95-475c-b441-04e91b3a8861"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:21.382079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.382046 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2" pod "b9176006-ca95-475c-b441-04e91b3a8861" (UID: "b9176006-ca95-475c-b441-04e91b3a8861"). InnerVolumeSpecName "kube-api-access-bsfj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:21.482073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.481982 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9176006-ca95-475c-b441-04e91b3a8861-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:21.482073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.482010 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsfj2\" (UniqueName: \"kubernetes.io/projected/b9176006-ca95-475c-b441-04e91b3a8861-kube-api-access-bsfj2\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:21.697225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.697192 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:21.697815 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.697752 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:21.699136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.699094 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:52:21.943202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.943168 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" Nov 03 20:52:21.944505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.943178 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr" event={"ID":"c00eeb35-be75-470f-82ce-85c54a7b117d","Type":"ContainerDied","Data":"b0e17fa85bc696e8a339c2d896588bf4dff208660fcecd9a5869744d4f5615d8"} Nov 03 20:52:21.944828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.944808 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0e17fa85bc696e8a339c2d896588bf4dff208660fcecd9a5869744d4f5615d8" Nov 03 20:52:21.949606 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.948162 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" event={"ID":"b9176006-ca95-475c-b441-04e91b3a8861","Type":"ContainerDied","Data":"3e9070109785ac1b648337619c0798eafd1e16410afcb62958dbad700db0010a"} Nov 03 20:52:21.949606 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.948218 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9070109785ac1b648337619c0798eafd1e16410afcb62958dbad700db0010a" Nov 03 20:52:21.950489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:21.950475 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z" Nov 03 20:52:22.086965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.086928 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379fcd8e-1f75-45d2-a5e5-5ba6e562d068" path="/var/lib/kubelet/pods/379fcd8e-1f75-45d2-a5e5-5ba6e562d068/volumes" Nov 03 20:52:22.155559 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.155529 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:22.292189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.292098 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca\") pod \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " Nov 03 20:52:22.292189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.292149 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd\" (UniqueName: \"kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd\") pod \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " Nov 03 20:52:22.292733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.292267 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca\") pod \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\" (UID: \"3ffdd307-0c75-4868-8f5f-f725b9b39c00\") " Nov 03 20:52:22.292733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.292546 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca" pod "3ffdd307-0c75-4868-8f5f-f725b9b39c00" (UID: "3ffdd307-0c75-4868-8f5f-f725b9b39c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:22.292812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.292787 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:22.294481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.294459 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca" pod "3ffdd307-0c75-4868-8f5f-f725b9b39c00" (UID: "3ffdd307-0c75-4868-8f5f-f725b9b39c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:22.294931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.294904 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd" pod "3ffdd307-0c75-4868-8f5f-f725b9b39c00" (UID: "3ffdd307-0c75-4868-8f5f-f725b9b39c00"). InnerVolumeSpecName "kube-api-access-6wdnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:22.393999 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.393966 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wdnd\" (UniqueName: \"kubernetes.io/projected/3ffdd307-0c75-4868-8f5f-f725b9b39c00-kube-api-access-6wdnd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:22.393999 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.393998 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ffdd307-0c75-4868-8f5f-f725b9b39c00-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:22.955860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.955831 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" Nov 03 20:52:22.956045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.955863 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl" event={"ID":"3ffdd307-0c75-4868-8f5f-f725b9b39c00","Type":"ContainerDied","Data":"2dafe91b3d60931422c96cd7e5145e4e881c8d2f9c3c74928ad935c57174a70e"} Nov 03 20:52:22.956045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:22.955941 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dafe91b3d60931422c96cd7e5145e4e881c8d2f9c3c74928ad935c57174a70e" Nov 03 20:52:23.782101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782067 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6"] Nov 03 20:52:23.782750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782728 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c00eeb35-be75-470f-82ce-85c54a7b117d" containerName="revoke-keys" Nov 03 20:52:23.782750 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782745 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c00eeb35-be75-470f-82ce-85c54a7b117d" containerName="revoke-keys" Nov 03 20:52:23.782860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782795 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b9176006-ca95-475c-b441-04e91b3a8861" containerName="revoke-keys" Nov 03 20:52:23.782860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782801 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9176006-ca95-475c-b441-04e91b3a8861" containerName="revoke-keys" Nov 03 20:52:23.782860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782808 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3ffdd307-0c75-4868-8f5f-f725b9b39c00" containerName="revoke-keys" Nov 03 20:52:23.782860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782814 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffdd307-0c75-4868-8f5f-f725b9b39c00" containerName="revoke-keys" Nov 03 20:52:23.782972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782911 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9176006-ca95-475c-b441-04e91b3a8861" containerName="revoke-keys" Nov 03 20:52:23.782972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782919 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffdd307-0c75-4868-8f5f-f725b9b39c00" containerName="revoke-keys" Nov 03 20:52:23.782972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.782927 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c00eeb35-be75-470f-82ce-85c54a7b117d" containerName="revoke-keys" Nov 03 20:52:23.786798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.786781 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:23.841278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.841241 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6"] Nov 03 20:52:23.881304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.881272 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:23.881506 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.881349 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:52:23.912852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.912822 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:23.912987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.912894 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksw5c\" (UniqueName: \"kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:23.913105 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:23.913090 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.014868 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.014829 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksw5c\" (UniqueName: \"kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.015090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.015070 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.015135 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.015120 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.015741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.015716 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.017450 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.017427 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.072899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.072831 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksw5c\" (UniqueName: \"kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c\") pod \"maas-api-revoke-sub-2a5df515-mgmq6\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.101611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.101579 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:24.539414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.539273 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6"] Nov 03 20:52:24.542816 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:24.542787 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc5dce0_f0f4_482f_bea9_97a728c74b00.slice/crio-543a896d25bd4391be9514d18b690cf993ef50180888e93a19430ed4d723c340 WatchSource:0}: Error finding container 543a896d25bd4391be9514d18b690cf993ef50180888e93a19430ed4d723c340: Status 404 returned error can't find the container with id 543a896d25bd4391be9514d18b690cf993ef50180888e93a19430ed4d723c340 Nov 03 20:52:24.974398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.974361 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" event={"ID":"5dc5dce0-f0f4-482f-bea9-97a728c74b00","Type":"ContainerStarted","Data":"f435236fad741a0952e4ea9091e4058908a616ae33161ab3188362c1e1764673"} Nov 03 20:52:24.974870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:24.974421 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" event={"ID":"5dc5dce0-f0f4-482f-bea9-97a728c74b00","Type":"ContainerStarted","Data":"543a896d25bd4391be9514d18b690cf993ef50180888e93a19430ed4d723c340"} Nov 03 20:52:25.086240 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.086129 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" podStartSLOduration=2.086114421 podStartE2EDuration="2.086114421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:25.08197447 +0000 UTC m=+1353.147215937" watchObservedRunningTime="2026-11-03 20:52:25.086114421 +0000 UTC m=+1353.151355894" Nov 03 20:52:25.708411 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.708330 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92"] Nov 03 20:52:25.713055 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.713032 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.828699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.826433 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92"] Nov 03 20:52:25.868904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.867104 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjh9\" (UniqueName: \"kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.868904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.867315 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.868904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.867412 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.968386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.968302 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.968386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.968373 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.968573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.968464 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjh9\" (UniqueName: \"kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.968975 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.968952 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.970753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.970731 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:25.984032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.984006 5932 generic.go:356] "Generic (PLEG): container finished" podID="5dc5dce0-f0f4-482f-bea9-97a728c74b00" containerID="f435236fad741a0952e4ea9091e4058908a616ae33161ab3188362c1e1764673" exitCode=0 Nov 03 20:52:25.984414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:25.984086 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" event={"ID":"5dc5dce0-f0f4-482f-bea9-97a728c74b00","Type":"ContainerDied","Data":"f435236fad741a0952e4ea9091e4058908a616ae33161ab3188362c1e1764673"} Nov 03 20:52:26.026416 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.026379 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjh9\" (UniqueName: \"kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9\") pod \"maas-api-revoke-sub-d1200aae-9kg92\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:26.028238 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.028220 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:26.413207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.413172 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92"] Nov 03 20:52:26.415889 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:26.415858 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0956623_54ef_4da1_9bad_0c7f1e07ff79.slice/crio-a754dff89c56c148e2b510666d6234f6ee2ac05c1754ae640d35bed6e7db448c WatchSource:0}: Error finding container a754dff89c56c148e2b510666d6234f6ee2ac05c1754ae640d35bed6e7db448c: Status 404 returned error can't find the container with id a754dff89c56c148e2b510666d6234f6ee2ac05c1754ae640d35bed6e7db448c Nov 03 20:52:26.510350 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.510304 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj"] Nov 03 20:52:26.519111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.519077 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.567799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.567763 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj"] Nov 03 20:52:26.576945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.576907 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.577131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.577042 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.577131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.577076 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwlx\" (UniqueName: \"kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.678523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.678433 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.678523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.678504 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.678760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.678529 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwlx\" (UniqueName: \"kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.679227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.679204 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.680960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.680939 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.739610 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.739581 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwlx\" (UniqueName: \"kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx\") pod \"maas-api-revoke-sub-4b501cf3-q49bj\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.833555 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.833510 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:26.994817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.994726 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" event={"ID":"f0956623-54ef-4da1-9bad-0c7f1e07ff79","Type":"ContainerStarted","Data":"91cbad3418d8ec155b7ff63c047a5ea6ca6cb509cda893371ca472406fd35303"} Nov 03 20:52:26.994817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:26.994789 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" event={"ID":"f0956623-54ef-4da1-9bad-0c7f1e07ff79","Type":"ContainerStarted","Data":"a754dff89c56c148e2b510666d6234f6ee2ac05c1754ae640d35bed6e7db448c"} Nov 03 20:52:27.109265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.109202 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" podStartSLOduration=2.109186831 podStartE2EDuration="2.109186831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:27.107796324 +0000 UTC m=+1355.173037797" watchObservedRunningTime="2026-11-03 20:52:27.109186831 +0000 UTC m=+1355.174428304" Nov 03 20:52:27.157388 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.157361 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:27.185491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.185458 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca\") pod \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " Nov 03 20:52:27.185690 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.185590 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c\" (UniqueName: \"kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c\") pod \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " Nov 03 20:52:27.185690 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.185652 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca\") pod \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\" (UID: \"5dc5dce0-f0f4-482f-bea9-97a728c74b00\") " Nov 03 20:52:27.186029 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.186004 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca" pod "5dc5dce0-f0f4-482f-bea9-97a728c74b00" (UID: "5dc5dce0-f0f4-482f-bea9-97a728c74b00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:27.187557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.187527 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca" pod "5dc5dce0-f0f4-482f-bea9-97a728c74b00" (UID: "5dc5dce0-f0f4-482f-bea9-97a728c74b00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:27.187882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.187845 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c" pod "5dc5dce0-f0f4-482f-bea9-97a728c74b00" (UID: "5dc5dce0-f0f4-482f-bea9-97a728c74b00"). InnerVolumeSpecName "kube-api-access-ksw5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:27.214540 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.214510 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj"] Nov 03 20:52:27.219206 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:27.219179 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfbca07b_8e25_4d0f_bbe3_025dc66edf16.slice/crio-15795a38df1dd166d7ab9e746b9b4dd352e3499c55db2576fe39b6e211f58943 WatchSource:0}: Error finding container 15795a38df1dd166d7ab9e746b9b4dd352e3499c55db2576fe39b6e211f58943: Status 404 returned error can't find the container with id 15795a38df1dd166d7ab9e746b9b4dd352e3499c55db2576fe39b6e211f58943 Nov 03 20:52:27.287469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.287438 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:27.287469 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.287468 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksw5c\" (UniqueName: \"kubernetes.io/projected/5dc5dce0-f0f4-482f-bea9-97a728c74b00-kube-api-access-ksw5c\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:27.287735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:27.287483 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dc5dce0-f0f4-482f-bea9-97a728c74b00-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:27.343793 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:52:27.343761 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0956623_54ef_4da1_9bad_0c7f1e07ff79.slice/crio-conmon-91cbad3418d8ec155b7ff63c047a5ea6ca6cb509cda893371ca472406fd35303.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:52:28.005038 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.005005 5932 generic.go:356] "Generic (PLEG): container finished" podID="f0956623-54ef-4da1-9bad-0c7f1e07ff79" containerID="91cbad3418d8ec155b7ff63c047a5ea6ca6cb509cda893371ca472406fd35303" exitCode=0 Nov 03 20:52:28.005501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.005097 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" event={"ID":"f0956623-54ef-4da1-9bad-0c7f1e07ff79","Type":"ContainerDied","Data":"91cbad3418d8ec155b7ff63c047a5ea6ca6cb509cda893371ca472406fd35303"} Nov 03 20:52:28.006819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.006790 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" event={"ID":"dfbca07b-8e25-4d0f-bbe3-025dc66edf16","Type":"ContainerStarted","Data":"0e5df1b9d715fd15690ea7d98cc4ff2292bc4950b8fdd07aa5321a352a063d93"} Nov 03 20:52:28.006950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.006842 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" event={"ID":"dfbca07b-8e25-4d0f-bbe3-025dc66edf16","Type":"ContainerStarted","Data":"15795a38df1dd166d7ab9e746b9b4dd352e3499c55db2576fe39b6e211f58943"} Nov 03 20:52:28.008087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.008068 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" Nov 03 20:52:28.008176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.008117 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6" event={"ID":"5dc5dce0-f0f4-482f-bea9-97a728c74b00","Type":"ContainerDied","Data":"543a896d25bd4391be9514d18b690cf993ef50180888e93a19430ed4d723c340"} Nov 03 20:52:28.008176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.008155 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543a896d25bd4391be9514d18b690cf993ef50180888e93a19430ed4d723c340" Nov 03 20:52:28.058389 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.058358 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn"] Nov 03 20:52:28.104244 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.104214 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc27b964-w4mhn"] Nov 03 20:52:28.292350 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.292252 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" podStartSLOduration=2.292237503 podStartE2EDuration="2.292237503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:28.288951416 +0000 UTC m=+1356.354192890" watchObservedRunningTime="2026-11-03 20:52:28.292237503 +0000 UTC m=+1356.357478975" Nov 03 20:52:28.298447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.298422 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5"] Nov 03 20:52:28.299139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.299125 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5dc5dce0-f0f4-482f-bea9-97a728c74b00" containerName="revoke-keys" Nov 03 20:52:28.299192 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.299140 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc5dce0-f0f4-482f-bea9-97a728c74b00" containerName="revoke-keys" Nov 03 20:52:28.299273 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.299261 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc5dce0-f0f4-482f-bea9-97a728c74b00" containerName="revoke-keys" Nov 03 20:52:28.303155 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.303132 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.354436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.354400 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5"] Nov 03 20:52:28.404525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.404489 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2w4j\" (UniqueName: \"kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.405292 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.405272 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.405416 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.405397 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.506869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.506833 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2w4j\" (UniqueName: \"kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.507177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.506946 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.507177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.507036 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.507872 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.507838 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.509428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.509402 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.564918 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.564836 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2w4j\" (UniqueName: \"kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j\") pod \"maas-api-revoke-sub-7e334666-d5xp5\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.618678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.618644 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:28.999632 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:28.999582 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5"] Nov 03 20:52:29.003261 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:29.003225 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd633a502_cde9_4c81_8fd1_5d26c1b7b120.slice/crio-7b8684cd4e54e4ec05a4861bd36194b74577be3a650fb5e757ce8256b866fbf2 WatchSource:0}: Error finding container 7b8684cd4e54e4ec05a4861bd36194b74577be3a650fb5e757ce8256b866fbf2: Status 404 returned error can't find the container with id 7b8684cd4e54e4ec05a4861bd36194b74577be3a650fb5e757ce8256b866fbf2 Nov 03 20:52:29.018525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.018495 5932 generic.go:356] "Generic (PLEG): container finished" podID="dfbca07b-8e25-4d0f-bbe3-025dc66edf16" containerID="0e5df1b9d715fd15690ea7d98cc4ff2292bc4950b8fdd07aa5321a352a063d93" exitCode=0 Nov 03 20:52:29.018937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.018594 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" event={"ID":"dfbca07b-8e25-4d0f-bbe3-025dc66edf16","Type":"ContainerDied","Data":"0e5df1b9d715fd15690ea7d98cc4ff2292bc4950b8fdd07aa5321a352a063d93"} Nov 03 20:52:29.020199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.020171 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" event={"ID":"d633a502-cde9-4c81-8fd1-5d26c1b7b120","Type":"ContainerStarted","Data":"7b8684cd4e54e4ec05a4861bd36194b74577be3a650fb5e757ce8256b866fbf2"} Nov 03 20:52:29.254182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.254115 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:29.317300 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.317269 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9\" (UniqueName: \"kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9\") pod \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " Nov 03 20:52:29.317484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.317317 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca\") pod \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " Nov 03 20:52:29.317484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.317412 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca\") pod \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\" (UID: \"f0956623-54ef-4da1-9bad-0c7f1e07ff79\") " Nov 03 20:52:29.317785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.317754 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca" pod "f0956623-54ef-4da1-9bad-0c7f1e07ff79" (UID: "f0956623-54ef-4da1-9bad-0c7f1e07ff79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:29.318062 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.318042 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:29.319479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.319453 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca" pod "f0956623-54ef-4da1-9bad-0c7f1e07ff79" (UID: "f0956623-54ef-4da1-9bad-0c7f1e07ff79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:29.319588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.319566 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9" pod "f0956623-54ef-4da1-9bad-0c7f1e07ff79" (UID: "f0956623-54ef-4da1-9bad-0c7f1e07ff79"). InnerVolumeSpecName "kube-api-access-lbjh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:29.419034 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.419001 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0956623-54ef-4da1-9bad-0c7f1e07ff79-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:29.419199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.419040 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbjh9\" (UniqueName: \"kubernetes.io/projected/f0956623-54ef-4da1-9bad-0c7f1e07ff79-kube-api-access-lbjh9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:29.868347 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.868309 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4"] Nov 03 20:52:29.869018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.869002 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f0956623-54ef-4da1-9bad-0c7f1e07ff79" containerName="revoke-keys" Nov 03 20:52:29.869086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.869020 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0956623-54ef-4da1-9bad-0c7f1e07ff79" containerName="revoke-keys" Nov 03 20:52:29.869170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.869161 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0956623-54ef-4da1-9bad-0c7f1e07ff79" containerName="revoke-keys" Nov 03 20:52:29.873022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.872995 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:29.927248 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.927213 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4"] Nov 03 20:52:29.927436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.927267 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:29.927579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.927553 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:29.927675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:29.927661 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8h6\" (UniqueName: \"kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.029235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.029198 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.029777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.029298 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8h6\" (UniqueName: \"kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.029777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.029366 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.030104 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.030084 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.030614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.030583 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" event={"ID":"d633a502-cde9-4c81-8fd1-5d26c1b7b120","Type":"ContainerStarted","Data":"8e1ecfad45f88d2cadbfcf059bc2c300d3a7e859f6aeb16aff2e78d4687d3168"} Nov 03 20:52:30.032035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.032014 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.032307 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.032285 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" Nov 03 20:52:30.032368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.032330 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92" event={"ID":"f0956623-54ef-4da1-9bad-0c7f1e07ff79","Type":"ContainerDied","Data":"a754dff89c56c148e2b510666d6234f6ee2ac05c1754ae640d35bed6e7db448c"} Nov 03 20:52:30.032404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.032373 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a754dff89c56c148e2b510666d6234f6ee2ac05c1754ae640d35bed6e7db448c" Nov 03 20:52:30.055067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.055033 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc2914e-7eb4-4375-a223-83854e55fd08" path="/var/lib/kubelet/pods/adc2914e-7eb4-4375-a223-83854e55fd08/volumes" Nov 03 20:52:30.086922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.086895 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8h6\" (UniqueName: \"kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6\") pod \"maas-api-revoke-sub-d8ec1550-gtsh4\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.146386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.146325 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" podStartSLOduration=2.146309761 podStartE2EDuration="2.146309761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:30.142786332 +0000 UTC m=+1358.208027801" watchObservedRunningTime="2026-11-03 20:52:30.146309761 +0000 UTC m=+1358.211551257" Nov 03 20:52:30.188101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.188078 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:30.204595 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.204567 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:30.332218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.332178 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca\") pod \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " Nov 03 20:52:30.332453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.332432 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca\") pod \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " Nov 03 20:52:30.332574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.332556 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx\" (UniqueName: \"kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx\") pod \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\" (UID: \"dfbca07b-8e25-4d0f-bbe3-025dc66edf16\") " Nov 03 20:52:30.332657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.332603 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca" pod "dfbca07b-8e25-4d0f-bbe3-025dc66edf16" (UID: "dfbca07b-8e25-4d0f-bbe3-025dc66edf16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:30.333112 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.333086 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.334777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.334745 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca" pod "dfbca07b-8e25-4d0f-bbe3-025dc66edf16" (UID: "dfbca07b-8e25-4d0f-bbe3-025dc66edf16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:30.335240 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.335211 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx" pod "dfbca07b-8e25-4d0f-bbe3-025dc66edf16" (UID: "dfbca07b-8e25-4d0f-bbe3-025dc66edf16"). InnerVolumeSpecName "kube-api-access-wtwlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:30.446364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.440776 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.446364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.440813 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtwlx\" (UniqueName: \"kubernetes.io/projected/dfbca07b-8e25-4d0f-bbe3-025dc66edf16-kube-api-access-wtwlx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:30.572894 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:30.572856 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4"] Nov 03 20:52:30.576512 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:30.576479 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e75b1e_3047_4156_9827_c05450b4476f.slice/crio-fd9246358661067e891eeef4b940b63438bf44e5502c621a1af1e87c1ef8a013 WatchSource:0}: Error finding container fd9246358661067e891eeef4b940b63438bf44e5502c621a1af1e87c1ef8a013: Status 404 returned error can't find the container with id fd9246358661067e891eeef4b940b63438bf44e5502c621a1af1e87c1ef8a013 Nov 03 20:52:31.042846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.042743 5932 generic.go:356] "Generic (PLEG): container finished" podID="d633a502-cde9-4c81-8fd1-5d26c1b7b120" containerID="8e1ecfad45f88d2cadbfcf059bc2c300d3a7e859f6aeb16aff2e78d4687d3168" exitCode=0 Nov 03 20:52:31.042846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.042828 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" event={"ID":"d633a502-cde9-4c81-8fd1-5d26c1b7b120","Type":"ContainerDied","Data":"8e1ecfad45f88d2cadbfcf059bc2c300d3a7e859f6aeb16aff2e78d4687d3168"} Nov 03 20:52:31.045132 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.045073 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" event={"ID":"e9e75b1e-3047-4156-9827-c05450b4476f","Type":"ContainerStarted","Data":"746da95743988880b77fb9644a22d3f6e4478df6420a4d82d777abcf94e06a2e"} Nov 03 20:52:31.045278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.045155 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" event={"ID":"e9e75b1e-3047-4156-9827-c05450b4476f","Type":"ContainerStarted","Data":"fd9246358661067e891eeef4b940b63438bf44e5502c621a1af1e87c1ef8a013"} Nov 03 20:52:31.046676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.046648 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" event={"ID":"dfbca07b-8e25-4d0f-bbe3-025dc66edf16","Type":"ContainerDied","Data":"15795a38df1dd166d7ab9e746b9b4dd352e3499c55db2576fe39b6e211f58943"} Nov 03 20:52:31.046754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.046684 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj" Nov 03 20:52:31.046754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.046699 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15795a38df1dd166d7ab9e746b9b4dd352e3499c55db2576fe39b6e211f58943" Nov 03 20:52:31.313107 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:31.313012 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" podStartSLOduration=2.312997413 podStartE2EDuration="2.312997413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:31.310076214 +0000 UTC m=+1359.375317690" watchObservedRunningTime="2026-11-03 20:52:31.312997413 +0000 UTC m=+1359.378238885" Nov 03 20:52:32.058091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.058061 5932 generic.go:356] "Generic (PLEG): container finished" podID="e9e75b1e-3047-4156-9827-c05450b4476f" containerID="746da95743988880b77fb9644a22d3f6e4478df6420a4d82d777abcf94e06a2e" exitCode=0 Nov 03 20:52:32.058580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.058149 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" event={"ID":"e9e75b1e-3047-4156-9827-c05450b4476f","Type":"ContainerDied","Data":"746da95743988880b77fb9644a22d3f6e4478df6420a4d82d777abcf94e06a2e"} Nov 03 20:52:32.229239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.229205 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:32.263787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.263757 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j\" (UniqueName: \"kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j\") pod \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " Nov 03 20:52:32.263949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.263931 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca\") pod \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " Nov 03 20:52:32.264026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.263970 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca\") pod \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\" (UID: \"d633a502-cde9-4c81-8fd1-5d26c1b7b120\") " Nov 03 20:52:32.264366 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.264332 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca" pod "d633a502-cde9-4c81-8fd1-5d26c1b7b120" (UID: "d633a502-cde9-4c81-8fd1-5d26c1b7b120"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:32.264674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.264651 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:32.265951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.265925 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca" pod "d633a502-cde9-4c81-8fd1-5d26c1b7b120" (UID: "d633a502-cde9-4c81-8fd1-5d26c1b7b120"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:32.266061 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.266050 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j" pod "d633a502-cde9-4c81-8fd1-5d26c1b7b120" (UID: "d633a502-cde9-4c81-8fd1-5d26c1b7b120"). InnerVolumeSpecName "kube-api-access-g2w4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:32.366418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.366319 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2w4j\" (UniqueName: \"kubernetes.io/projected/d633a502-cde9-4c81-8fd1-5d26c1b7b120-kube-api-access-g2w4j\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:32.366418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:32.366357 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d633a502-cde9-4c81-8fd1-5d26c1b7b120-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:33.068740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.068649 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" event={"ID":"d633a502-cde9-4c81-8fd1-5d26c1b7b120","Type":"ContainerDied","Data":"7b8684cd4e54e4ec05a4861bd36194b74577be3a650fb5e757ce8256b866fbf2"} Nov 03 20:52:33.068740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.068718 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8684cd4e54e4ec05a4861bd36194b74577be3a650fb5e757ce8256b866fbf2" Nov 03 20:52:33.068740 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.068656 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5" Nov 03 20:52:33.230069 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.230041 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:33.276897 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.276858 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6\" (UniqueName: \"kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6\") pod \"e9e75b1e-3047-4156-9827-c05450b4476f\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " Nov 03 20:52:33.277049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.276974 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca\") pod \"e9e75b1e-3047-4156-9827-c05450b4476f\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " Nov 03 20:52:33.277049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.277011 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca\") pod \"e9e75b1e-3047-4156-9827-c05450b4476f\" (UID: \"e9e75b1e-3047-4156-9827-c05450b4476f\") " Nov 03 20:52:33.277414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.277385 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca" pod "e9e75b1e-3047-4156-9827-c05450b4476f" (UID: "e9e75b1e-3047-4156-9827-c05450b4476f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:33.277708 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.277687 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:33.279212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.279184 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6" pod "e9e75b1e-3047-4156-9827-c05450b4476f" (UID: "e9e75b1e-3047-4156-9827-c05450b4476f"). InnerVolumeSpecName "kube-api-access-ts8h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:33.279376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.279353 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca" pod "e9e75b1e-3047-4156-9827-c05450b4476f" (UID: "e9e75b1e-3047-4156-9827-c05450b4476f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:33.379247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.379213 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts8h6\" (UniqueName: \"kubernetes.io/projected/e9e75b1e-3047-4156-9827-c05450b4476f-kube-api-access-ts8h6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:33.379383 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:33.379253 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9e75b1e-3047-4156-9827-c05450b4476f-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:34.079017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.078990 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" Nov 03 20:52:34.079368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.079021 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4" event={"ID":"e9e75b1e-3047-4156-9827-c05450b4476f","Type":"ContainerDied","Data":"fd9246358661067e891eeef4b940b63438bf44e5502c621a1af1e87c1ef8a013"} Nov 03 20:52:34.079368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.079073 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd9246358661067e891eeef4b940b63438bf44e5502c621a1af1e87c1ef8a013" Nov 03 20:52:34.983492 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.983453 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf"] Nov 03 20:52:34.984232 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984211 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e9e75b1e-3047-4156-9827-c05450b4476f" containerName="revoke-keys" Nov 03 20:52:34.984348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984233 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9e75b1e-3047-4156-9827-c05450b4476f" containerName="revoke-keys" Nov 03 20:52:34.984348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984263 5932 cpu_manager.go:414] "containerMap: removing container" podUID="dfbca07b-8e25-4d0f-bbe3-025dc66edf16" containerName="revoke-keys" Nov 03 20:52:34.984348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984272 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfbca07b-8e25-4d0f-bbe3-025dc66edf16" containerName="revoke-keys" Nov 03 20:52:34.984348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984285 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d633a502-cde9-4c81-8fd1-5d26c1b7b120" containerName="revoke-keys" Nov 03 20:52:34.984348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984294 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d633a502-cde9-4c81-8fd1-5d26c1b7b120" containerName="revoke-keys" Nov 03 20:52:34.984596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984467 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9e75b1e-3047-4156-9827-c05450b4476f" containerName="revoke-keys" Nov 03 20:52:34.984596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984485 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfbca07b-8e25-4d0f-bbe3-025dc66edf16" containerName="revoke-keys" Nov 03 20:52:34.984596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.984496 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d633a502-cde9-4c81-8fd1-5d26c1b7b120" containerName="revoke-keys" Nov 03 20:52:34.988473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:34.988449 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.048012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.047964 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf"] Nov 03 20:52:35.082453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.082411 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:35.082973 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.082537 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:52:35.101761 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.101732 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s"] Nov 03 20:52:35.106256 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.106227 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b3ba55-9ct5s"] Nov 03 20:52:35.108122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.108087 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.108237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.108149 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7fc\" (UniqueName: \"kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.108316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.108303 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.209290 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.209247 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.209446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.209300 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7fc\" (UniqueName: \"kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.209446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.209394 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.209932 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.209908 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.211806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.211784 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.267229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.267156 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7fc\" (UniqueName: \"kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc\") pod \"maas-api-revoke-sub-fe1fdc9d-tkjvf\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.303266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.303234 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:35.691505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:35.691472 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf"] Nov 03 20:52:35.695305 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:35.695270 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380fdb76_75fc_4ea5_a3a8_2e995140667a.slice/crio-32edcd71b64a4a7272c896c26a696cba08880656decf08e4f097121fcbb75fda WatchSource:0}: Error finding container 32edcd71b64a4a7272c896c26a696cba08880656decf08e4f097121fcbb75fda: Status 404 returned error can't find the container with id 32edcd71b64a4a7272c896c26a696cba08880656decf08e4f097121fcbb75fda Nov 03 20:52:36.053465 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:36.053437 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a837079-380b-4a95-b7a8-3bfbb15d5609" path="/var/lib/kubelet/pods/2a837079-380b-4a95-b7a8-3bfbb15d5609/volumes" Nov 03 20:52:36.102206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:36.102172 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" event={"ID":"380fdb76-75fc-4ea5-a3a8-2e995140667a","Type":"ContainerStarted","Data":"547f8f347f1c7f927b861f75e197273115b633319ab2a172ad7057415b0c1027"} Nov 03 20:52:36.102592 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:36.102231 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" event={"ID":"380fdb76-75fc-4ea5-a3a8-2e995140667a","Type":"ContainerStarted","Data":"32edcd71b64a4a7272c896c26a696cba08880656decf08e4f097121fcbb75fda"} Nov 03 20:52:36.210868 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:36.210817 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" podStartSLOduration=2.21080298 podStartE2EDuration="2.21080298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:36.209939999 +0000 UTC m=+1364.275181472" watchObservedRunningTime="2026-11-03 20:52:36.21080298 +0000 UTC m=+1364.276044452" Nov 03 20:52:37.112725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.112641 5932 generic.go:356] "Generic (PLEG): container finished" podID="380fdb76-75fc-4ea5-a3a8-2e995140667a" containerID="547f8f347f1c7f927b861f75e197273115b633319ab2a172ad7057415b0c1027" exitCode=0 Nov 03 20:52:37.112725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.112683 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" event={"ID":"380fdb76-75fc-4ea5-a3a8-2e995140667a","Type":"ContainerDied","Data":"547f8f347f1c7f927b861f75e197273115b633319ab2a172ad7057415b0c1027"} Nov 03 20:52:37.212884 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.212849 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8"] Nov 03 20:52:37.217448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.217430 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.268305 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.268272 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8"] Nov 03 20:52:37.339796 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.339760 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.339964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.339916 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.340008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.339969 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znz5d\" (UniqueName: \"kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.441070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.441025 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.441246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.441082 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znz5d\" (UniqueName: \"kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.441285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.441252 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.441826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.441808 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.443700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.443681 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.498675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.498614 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znz5d\" (UniqueName: \"kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d\") pod \"maas-api-revoke-sub-98dcbc6a-98fc8\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.532464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.532434 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:37.925066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:37.925032 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8"] Nov 03 20:52:38.122818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.122782 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" event={"ID":"a9246355-4a0a-4a2d-9616-e032bde5dffd","Type":"ContainerStarted","Data":"bc53a59175ae18ee0f21a172c5d37d21b8f1f03fbb3977c812c1f4eaf999e2e5"} Nov 03 20:52:38.260809 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.260783 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:38.354432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.354406 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc\" (UniqueName: \"kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc\") pod \"380fdb76-75fc-4ea5-a3a8-2e995140667a\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " Nov 03 20:52:38.354596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.354584 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca\") pod \"380fdb76-75fc-4ea5-a3a8-2e995140667a\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " Nov 03 20:52:38.354692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.354635 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca\") pod \"380fdb76-75fc-4ea5-a3a8-2e995140667a\" (UID: \"380fdb76-75fc-4ea5-a3a8-2e995140667a\") " Nov 03 20:52:38.354945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.354916 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca" pod "380fdb76-75fc-4ea5-a3a8-2e995140667a" (UID: "380fdb76-75fc-4ea5-a3a8-2e995140667a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:38.355044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.355030 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:38.356567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.356542 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca" pod "380fdb76-75fc-4ea5-a3a8-2e995140667a" (UID: "380fdb76-75fc-4ea5-a3a8-2e995140667a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:38.356675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.356575 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc" pod "380fdb76-75fc-4ea5-a3a8-2e995140667a" (UID: "380fdb76-75fc-4ea5-a3a8-2e995140667a"). InnerVolumeSpecName "kube-api-access-sl7fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:38.456871 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.456840 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/380fdb76-75fc-4ea5-a3a8-2e995140667a-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:38.457000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:38.456873 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl7fc\" (UniqueName: \"kubernetes.io/projected/380fdb76-75fc-4ea5-a3a8-2e995140667a-kube-api-access-sl7fc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:39.133130 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.133046 5932 generic.go:356] "Generic (PLEG): container finished" podID="a9246355-4a0a-4a2d-9616-e032bde5dffd" containerID="b4944c06b3ca4b1e4b0221c7cf4d7e32309c07a8552546f1f19d1a4f64ebbc66" exitCode=0 Nov 03 20:52:39.133543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.133133 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" event={"ID":"a9246355-4a0a-4a2d-9616-e032bde5dffd","Type":"ContainerDied","Data":"b4944c06b3ca4b1e4b0221c7cf4d7e32309c07a8552546f1f19d1a4f64ebbc66"} Nov 03 20:52:39.134647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.134603 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" Nov 03 20:52:39.134647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.134639 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf" event={"ID":"380fdb76-75fc-4ea5-a3a8-2e995140667a","Type":"ContainerDied","Data":"32edcd71b64a4a7272c896c26a696cba08880656decf08e4f097121fcbb75fda"} Nov 03 20:52:39.134786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.134677 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32edcd71b64a4a7272c896c26a696cba08880656decf08e4f097121fcbb75fda" Nov 03 20:52:39.981000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.980966 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh"] Nov 03 20:52:39.981765 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.981744 5932 cpu_manager.go:414] "containerMap: removing container" podUID="380fdb76-75fc-4ea5-a3a8-2e995140667a" containerName="revoke-keys" Nov 03 20:52:39.981765 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.981765 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="380fdb76-75fc-4ea5-a3a8-2e995140667a" containerName="revoke-keys" Nov 03 20:52:39.981985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.981971 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="380fdb76-75fc-4ea5-a3a8-2e995140667a" containerName="revoke-keys" Nov 03 20:52:39.985767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:39.985745 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.040494 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.040450 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh"] Nov 03 20:52:40.078298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.078255 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.078464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.078413 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.078464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.078441 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4bd\" (UniqueName: \"kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.180018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.179855 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.180018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.179996 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.180018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.180021 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4bd\" (UniqueName: \"kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.180707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.180688 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.182976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.182949 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.238338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.238257 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4bd\" (UniqueName: \"kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd\") pod \"maas-api-revoke-sub-885e9d89-gkbvh\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.299772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.299740 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:40.310902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.310879 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:40.382217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.382183 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d\" (UniqueName: \"kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d\") pod \"a9246355-4a0a-4a2d-9616-e032bde5dffd\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " Nov 03 20:52:40.382353 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.382286 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca\") pod \"a9246355-4a0a-4a2d-9616-e032bde5dffd\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " Nov 03 20:52:40.382501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.382486 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca\") pod \"a9246355-4a0a-4a2d-9616-e032bde5dffd\" (UID: \"a9246355-4a0a-4a2d-9616-e032bde5dffd\") " Nov 03 20:52:40.383635 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.383599 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca" pod "a9246355-4a0a-4a2d-9616-e032bde5dffd" (UID: "a9246355-4a0a-4a2d-9616-e032bde5dffd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:40.386156 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.386129 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca" pod "a9246355-4a0a-4a2d-9616-e032bde5dffd" (UID: "a9246355-4a0a-4a2d-9616-e032bde5dffd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:40.386264 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.386207 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d" pod "a9246355-4a0a-4a2d-9616-e032bde5dffd" (UID: "a9246355-4a0a-4a2d-9616-e032bde5dffd"). InnerVolumeSpecName "kube-api-access-znz5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:40.484030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.484000 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:40.484030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.484026 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znz5d\" (UniqueName: \"kubernetes.io/projected/a9246355-4a0a-4a2d-9616-e032bde5dffd-kube-api-access-znz5d\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:40.484030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.484036 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9246355-4a0a-4a2d-9616-e032bde5dffd-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:40.684405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:40.684369 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh"] Nov 03 20:52:40.688454 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:40.688424 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdba8e76_bfc3_42d8_b4ee_cc722124f5a4.slice/crio-737720d666e461cac15c5a3e58084cf2b4aae56657065eb4f0ed3e5ed9a4727c WatchSource:0}: Error finding container 737720d666e461cac15c5a3e58084cf2b4aae56657065eb4f0ed3e5ed9a4727c: Status 404 returned error can't find the container with id 737720d666e461cac15c5a3e58084cf2b4aae56657065eb4f0ed3e5ed9a4727c Nov 03 20:52:41.153826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:41.153780 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" event={"ID":"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4","Type":"ContainerStarted","Data":"5ce3dc355689e7c72bf155c51f2f986d663835c00e8daccb0541da1162344d91"} Nov 03 20:52:41.154008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:41.153844 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" event={"ID":"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4","Type":"ContainerStarted","Data":"737720d666e461cac15c5a3e58084cf2b4aae56657065eb4f0ed3e5ed9a4727c"} Nov 03 20:52:41.155332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:41.155309 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" Nov 03 20:52:41.155446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:41.155310 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8" event={"ID":"a9246355-4a0a-4a2d-9616-e032bde5dffd","Type":"ContainerDied","Data":"bc53a59175ae18ee0f21a172c5d37d21b8f1f03fbb3977c812c1f4eaf999e2e5"} Nov 03 20:52:41.155446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:41.155434 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc53a59175ae18ee0f21a172c5d37d21b8f1f03fbb3977c812c1f4eaf999e2e5" Nov 03 20:52:41.263033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:41.262973 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" podStartSLOduration=2.262956866 podStartE2EDuration="2.262956866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:41.261955795 +0000 UTC m=+1369.327197271" watchObservedRunningTime="2026-11-03 20:52:41.262956866 +0000 UTC m=+1369.328198338" Nov 03 20:52:42.169890 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.169851 5932 generic.go:356] "Generic (PLEG): container finished" podID="fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" containerID="5ce3dc355689e7c72bf155c51f2f986d663835c00e8daccb0541da1162344d91" exitCode=0 Nov 03 20:52:42.170101 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.169935 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" event={"ID":"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4","Type":"ContainerDied","Data":"5ce3dc355689e7c72bf155c51f2f986d663835c00e8daccb0541da1162344d91"} Nov 03 20:52:42.613333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.613249 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb"] Nov 03 20:52:42.613948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.613933 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a9246355-4a0a-4a2d-9616-e032bde5dffd" containerName="revoke-keys" Nov 03 20:52:42.614008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.613949 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9246355-4a0a-4a2d-9616-e032bde5dffd" containerName="revoke-keys" Nov 03 20:52:42.614047 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.614042 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9246355-4a0a-4a2d-9616-e032bde5dffd" containerName="revoke-keys" Nov 03 20:52:42.617855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.617838 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724071 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724039 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724127 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724188 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724389 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724363 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724445 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724474 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724517 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724502 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724571 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.724675 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.724600 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdsm\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.730554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.730523 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb"] Nov 03 20:52:42.825441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825408 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825441 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825445 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdsm\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825518 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825592 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825662 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825760 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825824 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.825853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825848 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.826017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825879 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.826017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.825877 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.826124 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.826046 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.826305 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.826283 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.826405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.826379 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.826470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.826456 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.827986 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.827960 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.828330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.828311 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.882921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.882884 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.883138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.883115 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdsm\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:42.931497 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:42.931457 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:43.315455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.315427 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:43.369608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.369576 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb"] Nov 03 20:52:43.373407 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:43.373380 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c273ee_2cca_417b_a2be_4132c3a271be.slice/crio-02e80b8c3c9b5a2782c5e34db7bfd27025516ecf23476d56cd745daab21e36c9 WatchSource:0}: Error finding container 02e80b8c3c9b5a2782c5e34db7bfd27025516ecf23476d56cd745daab21e36c9: Status 404 returned error can't find the container with id 02e80b8c3c9b5a2782c5e34db7bfd27025516ecf23476d56cd745daab21e36c9 Nov 03 20:52:43.376293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.376256 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:43.376375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.376345 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:43.376414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.376388 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:43.434467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.434435 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca\") pod \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " Nov 03 20:52:43.434648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.434605 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca\") pod \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " Nov 03 20:52:43.434725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.434646 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd\" (UniqueName: \"kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd\") pod \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\" (UID: \"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4\") " Nov 03 20:52:43.435839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.435814 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca" pod "fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" (UID: "fdba8e76-bfc3-42d8-b4ee-cc722124f5a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:43.437795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.437765 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca" pod "fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" (UID: "fdba8e76-bfc3-42d8-b4ee-cc722124f5a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:43.437908 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.437839 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd" pod "fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" (UID: "fdba8e76-bfc3-42d8-b4ee-cc722124f5a4"). InnerVolumeSpecName "kube-api-access-vj4bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:43.536550 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.536518 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:43.536550 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.536548 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4bd\" (UniqueName: \"kubernetes.io/projected/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-kube-api-access-vj4bd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:43.536550 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:43.536559 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:44.206893 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.206864 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" Nov 03 20:52:44.207657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.206955 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh" event={"ID":"fdba8e76-bfc3-42d8-b4ee-cc722124f5a4","Type":"ContainerDied","Data":"737720d666e461cac15c5a3e58084cf2b4aae56657065eb4f0ed3e5ed9a4727c"} Nov 03 20:52:44.207798 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.207671 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="737720d666e461cac15c5a3e58084cf2b4aae56657065eb4f0ed3e5ed9a4727c" Nov 03 20:52:44.209221 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.209194 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" event={"ID":"b8c273ee-2cca-417b-a2be-4132c3a271be","Type":"ContainerStarted","Data":"d275b1b10ba9158a1d007e3f7ed8226bb77fef3c1b2a1a7a4d0a3075a34ce285"} Nov 03 20:52:44.209391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.209372 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" event={"ID":"b8c273ee-2cca-417b-a2be-4132c3a271be","Type":"ContainerStarted","Data":"02e80b8c3c9b5a2782c5e34db7bfd27025516ecf23476d56cd745daab21e36c9"} Nov 03 20:52:44.211636 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:52:44.211604 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:52:44.375607 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.375534 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podStartSLOduration=2.375519232 podStartE2EDuration="2.375519232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:44.371063591 +0000 UTC m=+1372.436305068" watchObservedRunningTime="2026-11-03 20:52:44.375519232 +0000 UTC m=+1372.440760704" Nov 03 20:52:44.942647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.942024 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:44.945707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.945005 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:44.945707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:44.945083 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:45.701761 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.701728 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8"] Nov 03 20:52:45.702392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.702377 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" containerName="revoke-keys" Nov 03 20:52:45.702449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.702393 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" containerName="revoke-keys" Nov 03 20:52:45.702530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.702521 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" containerName="revoke-keys" Nov 03 20:52:45.706483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.706466 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:45.760980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.760946 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8"] Nov 03 20:52:45.786012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.785739 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9n4n\" (UniqueName: \"kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:45.786012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.785825 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:45.786012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.785911 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:45.818692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.805043 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:52:45.818692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.805129 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:45.943659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.936241 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:45.943659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:45.936337 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:46.068705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.065310 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.068705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.065558 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9n4n\" (UniqueName: \"kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.068705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.065653 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.068705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.066633 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.244649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.236882 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.514700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.507435 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9n4n\" (UniqueName: \"kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n\") pod \"maas-api-revoke-sub-0151c44b-rdlj8\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.622647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.620877 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:46.932995 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.932956 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:46.933584 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.933548 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:46.991764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:46.989732 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.307694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:47.304775 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.521650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:47.483063 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 20:52:47.826647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:47.816901 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:47.826647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:47.817905 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.078703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.077813 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.123647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.123327 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:48.123647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.123388 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:48.319647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.310790 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.319647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.315690 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:52:48.450835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.439211 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded" Nov 03 20:52:48.504791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.504742 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 20:52:48.554686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.529332 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8"] Nov 03 20:52:48.558645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.555043 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:52:48.766722 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.762903 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 20:52:48.793827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.793789 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:48.794168 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.794133 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.838825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.817721 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:48.838825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.818943 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:52:48.838825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:48.828876 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.142904 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.126132 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:52:49.210646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.197315 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:49.210646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.197374 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:49.210646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.199324 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.239118 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.239070 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.239759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.239729 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:52:49.427986 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.427875 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:52:49.550937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.550892 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.785202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.785106 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.785891 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.785869 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:52:49.872235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.872191 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:52:49.873837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.873803 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" event={"ID":"83f34b31-3fb8-4f26-83b2-a955443251eb","Type":"ContainerStarted","Data":"bea4d046a5f141e589cca17a819c63a140981df411833f927acb2624149e0c31"} Nov 03 20:52:49.874698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.874665 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:49.935601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.935563 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:49.936706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:49.936605 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:50.200646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.182771 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.201613 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.201287 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.378648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.376778 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.523731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.402902 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:52:50.523731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.517935 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:52:50.564488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.551070 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.686127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.686089 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g"] Nov 03 20:52:50.736444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.736407 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:50.736728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.736694 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.797711 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.797595 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.914268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.914203 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:50.916599 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.916565 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:52:50.936104 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.936071 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:50.936390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:50.936353 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:51.219406 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.218026 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.380819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.379485 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.531702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.530940 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:51.560714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.559738 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 20:52:51.636309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.636270 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57ccf053-c8n4g"] Nov 03 20:52:51.665649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.651258 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" event={"ID":"83f34b31-3fb8-4f26-83b2-a955443251eb","Type":"ContainerStarted","Data":"cb475fe150b974f53be294d51f194f4dbbc4d1a039cf40b8d83d40dd713e59ed"} Nov 03 20:52:51.814713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.807280 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 20:52:51.885648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.882114 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:51.885648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:51.882183 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.041659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.030981 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:52:52.045647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.043472 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.109707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.101803 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.109707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.102923 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:52.109707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.102985 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:52.241939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.241891 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.300146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.285176 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:52.300146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.285237 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.379705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.379671 5932 scope.go:122] "RemoveContainer" containerID="024913e2c361b962bb09a1c39cf563185a0dc941bb5afb45750acfc5664c0d2e" Nov 03 20:52:52.380470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.380333 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.533829 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.533782 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.560804 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.560735 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:52.650459 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.650371 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcc03fd-9fa9-41f4-a46c-48e637963102" path="/var/lib/kubelet/pods/cbcc03fd-9fa9-41f4-a46c-48e637963102/volumes" Nov 03 20:52:52.790235 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.790132 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:52:52.798149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.798108 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:52:52.803797 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.803768 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:52:52.851930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.841115 5932 scope.go:122] "RemoveContainer" containerID="425466fc2e646103fdf54872c4e31fd7bed335b6066f08740c07e6b0d0e97137" Nov 03 20:52:52.875814 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.875779 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:52:52.891236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.887485 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:52:52.891236 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.887980 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:52:52.899147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.899095 5932 generic.go:356] "Generic (PLEG): container finished" podID="83f34b31-3fb8-4f26-83b2-a955443251eb" containerID="cb475fe150b974f53be294d51f194f4dbbc4d1a039cf40b8d83d40dd713e59ed" exitCode=0 Nov 03 20:52:52.913229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.907787 5932 scope.go:122] "RemoveContainer" containerID="1d9550e87e4ba7a35d4c60ca426df0c55964a68c0904f94c4f781cf21f2f9b25" Nov 03 20:52:52.916841 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.908206 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" event={"ID":"83f34b31-3fb8-4f26-83b2-a955443251eb","Type":"ContainerDied","Data":"cb475fe150b974f53be294d51f194f4dbbc4d1a039cf40b8d83d40dd713e59ed"} Nov 03 20:52:52.917112 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.917096 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:52:52.938314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.938286 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:52.939202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.939174 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:52.984345 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.984309 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb"] Nov 03 20:52:52.989684 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.989659 5932 scope.go:122] "RemoveContainer" containerID="0ff6ef3c663b5930ca538282238a5131bcf5f74115fe63ea6f8e355faa6663a0" Nov 03 20:52:52.992957 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.992934 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb"] Nov 03 20:52:52.993075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:52.993022 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.003791 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.003768 5932 scope.go:122] "RemoveContainer" containerID="88e0f34eba9cdad3b53270f584f6bd636692d46caf6c93ca878bb401b67cfeb4" Nov 03 20:52:53.086500 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.085072 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.086500 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.085284 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.086500 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.085391 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xbz7\" (UniqueName: \"kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.187243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.187201 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xbz7\" (UniqueName: \"kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.187392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.187338 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.187505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.187486 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.188405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.188375 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.199067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.199027 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.220722 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.220683 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:53.261394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.261363 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xbz7\" (UniqueName: \"kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7\") pod \"maas-api-revoke-sub-adeeb386-s2dqb\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.314214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.314183 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:53.391913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.391084 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca\") pod \"83f34b31-3fb8-4f26-83b2-a955443251eb\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " Nov 03 20:52:53.391913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.391172 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n\" (UniqueName: \"kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n\") pod \"83f34b31-3fb8-4f26-83b2-a955443251eb\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " Nov 03 20:52:53.391913 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.391514 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca\") pod \"83f34b31-3fb8-4f26-83b2-a955443251eb\" (UID: \"83f34b31-3fb8-4f26-83b2-a955443251eb\") " Nov 03 20:52:53.392573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.392309 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca" pod "83f34b31-3fb8-4f26-83b2-a955443251eb" (UID: "83f34b31-3fb8-4f26-83b2-a955443251eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:53.392573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.392534 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:53.393908 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.393865 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca" pod "83f34b31-3fb8-4f26-83b2-a955443251eb" (UID: "83f34b31-3fb8-4f26-83b2-a955443251eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:53.394023 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.393929 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n" pod "83f34b31-3fb8-4f26-83b2-a955443251eb" (UID: "83f34b31-3fb8-4f26-83b2-a955443251eb"). InnerVolumeSpecName "kube-api-access-k9n4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:53.512055 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.512017 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f34b31-3fb8-4f26-83b2-a955443251eb-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:53.512202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.512067 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9n4n\" (UniqueName: \"kubernetes.io/projected/83f34b31-3fb8-4f26-83b2-a955443251eb-kube-api-access-k9n4n\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:53.790681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.790234 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb"] Nov 03 20:52:53.791777 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:53.791751 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a7b8d3_b8db_4aed_853e_4e05ab491cf8.slice/crio-19dfe3f9151ee18f10fd2ece32152cc7d953dc9c085d4d68469fce28d2b504b6 WatchSource:0}: Error finding container 19dfe3f9151ee18f10fd2ece32152cc7d953dc9c085d4d68469fce28d2b504b6: Status 404 returned error can't find the container with id 19dfe3f9151ee18f10fd2ece32152cc7d953dc9c085d4d68469fce28d2b504b6 Nov 03 20:52:53.945651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.942658 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 20:52:53.945651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.942720 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 20:52:53.951271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.951244 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" Nov 03 20:52:53.952768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.951261 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8" event={"ID":"83f34b31-3fb8-4f26-83b2-a955443251eb","Type":"ContainerDied","Data":"bea4d046a5f141e589cca17a819c63a140981df411833f927acb2624149e0c31"} Nov 03 20:52:53.952768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.951322 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bea4d046a5f141e589cca17a819c63a140981df411833f927acb2624149e0c31" Nov 03 20:52:53.962727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:53.959224 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" event={"ID":"97a7b8d3-b8db-4aed-853e-4e05ab491cf8","Type":"ContainerStarted","Data":"19dfe3f9151ee18f10fd2ece32152cc7d953dc9c085d4d68469fce28d2b504b6"} Nov 03 20:52:54.429952 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.429909 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn"] Nov 03 20:52:54.431085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.431057 5932 cpu_manager.go:414] "containerMap: removing container" podUID="83f34b31-3fb8-4f26-83b2-a955443251eb" containerName="revoke-keys" Nov 03 20:52:54.431085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.431084 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f34b31-3fb8-4f26-83b2-a955443251eb" containerName="revoke-keys" Nov 03 20:52:54.431277 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.431265 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f34b31-3fb8-4f26-83b2-a955443251eb" containerName="revoke-keys" Nov 03 20:52:54.435963 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.435941 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.495057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.495025 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn"] Nov 03 20:52:54.536400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.536364 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.536552 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.536478 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.536598 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.536567 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2smn\" (UniqueName: \"kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.638531 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.638494 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2smn\" (UniqueName: \"kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.638707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.638631 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.638755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.638714 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.639346 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.639314 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.640870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.640844 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.696751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.696672 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2smn\" (UniqueName: \"kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn\") pod \"maas-api-revoke-sub-4953da8e-rrsbn\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.752736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.752704 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:54.971253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.971165 5932 generic.go:356] "Generic (PLEG): container finished" podID="97a7b8d3-b8db-4aed-853e-4e05ab491cf8" containerID="733da3ca8cbab3a5c199ee60f4e317d8f7ee6ce715b4a64a6ab8b2317c5d91da" exitCode=0 Nov 03 20:52:54.971391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:54.971317 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" event={"ID":"97a7b8d3-b8db-4aed-853e-4e05ab491cf8","Type":"ContainerDied","Data":"733da3ca8cbab3a5c199ee60f4e317d8f7ee6ce715b4a64a6ab8b2317c5d91da"} Nov 03 20:52:55.208998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.208715 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn"] Nov 03 20:52:55.213681 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:55.212049 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2091b247_7369_41a1_9559_3ce1defa7538.slice/crio-dc9fc41ca9d4167be097de2f4669d507ea2934c6925fea75880321ef0286a7c8 WatchSource:0}: Error finding container dc9fc41ca9d4167be097de2f4669d507ea2934c6925fea75880321ef0286a7c8: Status 404 returned error can't find the container with id dc9fc41ca9d4167be097de2f4669d507ea2934c6925fea75880321ef0286a7c8 Nov 03 20:52:55.214878 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.214484 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9"] Nov 03 20:52:55.221238 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.221216 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.258400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.258366 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.258528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.258420 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vngl\" (UniqueName: \"kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.258652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.258611 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.273067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.273037 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9"] Nov 03 20:52:55.283341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.283309 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr"] Nov 03 20:52:55.288701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.288680 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.340899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.339120 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-b6eaffa8-gw-openshift-default-dockercfg-45jgb\"" Nov 03 20:52:55.359610 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.359567 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr"] Nov 03 20:52:55.361370 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361336 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.361502 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361396 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.361502 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361432 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.361639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361518 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.361639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361552 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.361639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361634 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.361792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361750 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.361846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.361786 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vngl\" (UniqueName: \"kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.362742 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.362718 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.363073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.363047 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfjfv\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.363172 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.363109 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.363172 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.363139 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.363339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.363172 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.439326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.438082 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.487854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.487754 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.488109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.488077 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.488244 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.488229 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.488462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.488425 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.488582 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.488566 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.488782 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.488747 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.489063 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.489044 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfjfv\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.489203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.489189 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.489344 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.489308 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.490051 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.489991 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.490449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.490429 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.491135 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.491101 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.492115 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.491555 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.492938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.492918 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.493213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.493167 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vngl\" (UniqueName: \"kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl\") pod \"maas-api-revoke-sub-1a3cb6a7-7hzd9\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.493876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.493853 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.536428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.535061 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.613644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.595322 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:55.615976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.615899 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.658728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.637362 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfjfv\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv\") pod \"e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.907216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.906893 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:55.957658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.944646 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:55.957658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:55.944777 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:56.013795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.013744 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" event={"ID":"2091b247-7369-41a1-9559-3ce1defa7538","Type":"ContainerStarted","Data":"2427681c6d40b80b612c7a951c7001f1e9db0840fac1c047e3fabcd74f893bc3"} Nov 03 20:52:56.013891 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.013814 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" event={"ID":"2091b247-7369-41a1-9559-3ce1defa7538","Type":"ContainerStarted","Data":"dc9fc41ca9d4167be097de2f4669d507ea2934c6925fea75880321ef0286a7c8"} Nov 03 20:52:56.074948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.074913 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2"] Nov 03 20:52:56.087965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.087932 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179388 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179468 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179532 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179589 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179713 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179785 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgnbm\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179902 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179947 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.179977 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.265707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.260005 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-15564eb1-openshift-default-dockercfg-z76q5\"" Nov 03 20:52:56.479136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.479101 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2"] Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.490500 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.490600 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.490689 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.490791 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.490845 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.490935 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.491011 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.491176 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.491293 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgnbm\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.493655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.493206 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.522404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.508792 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.522404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.509115 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.522404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.509381 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.527923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.527896 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.649788 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.643804 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" podStartSLOduration=2.643782587 podStartE2EDuration="2.643782587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:56.136331519 +0000 UTC m=+1384.201572994" watchObservedRunningTime="2026-11-03 20:52:56.643782587 +0000 UTC m=+1384.709024062" Nov 03 20:52:56.666208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.665968 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9"] Nov 03 20:52:56.670221 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.670188 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.689488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.689431 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:56.689728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.689581 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:56.689728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.689683 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:56.690408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.690377 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr"] Nov 03 20:52:56.696316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.696291 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgnbm\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.699990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.699959 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.700215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.700191 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy\") pod \"e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:56.727969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.727943 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:56.813348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.813253 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca\") pod \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " Nov 03 20:52:56.813348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.813313 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7\" (UniqueName: \"kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7\") pod \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " Nov 03 20:52:56.813590 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.813393 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca\") pod \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\" (UID: \"97a7b8d3-b8db-4aed-853e-4e05ab491cf8\") " Nov 03 20:52:56.817759 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.817724 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca" pod "97a7b8d3-b8db-4aed-853e-4e05ab491cf8" (UID: "97a7b8d3-b8db-4aed-853e-4e05ab491cf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:56.820741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.820711 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7" pod "97a7b8d3-b8db-4aed-853e-4e05ab491cf8" (UID: "97a7b8d3-b8db-4aed-853e-4e05ab491cf8"). InnerVolumeSpecName "kube-api-access-4xbz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:56.822843 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.822803 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca" pod "97a7b8d3-b8db-4aed-853e-4e05ab491cf8" (UID: "97a7b8d3-b8db-4aed-853e-4e05ab491cf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:56.943646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.939195 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.943646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.939254 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.943646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.939272 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xbz7\" (UniqueName: \"kubernetes.io/projected/97a7b8d3-b8db-4aed-853e-4e05ab491cf8-kube-api-access-4xbz7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.993942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.992908 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:52:56.994363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.994235 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:52:57.001456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:56.993327 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:57.033907 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.033773 5932 generic.go:356] "Generic (PLEG): container finished" podID="2091b247-7369-41a1-9559-3ce1defa7538" containerID="2427681c6d40b80b612c7a951c7001f1e9db0840fac1c047e3fabcd74f893bc3" exitCode=0 Nov 03 20:52:57.034376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.034341 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" event={"ID":"2091b247-7369-41a1-9559-3ce1defa7538","Type":"ContainerDied","Data":"2427681c6d40b80b612c7a951c7001f1e9db0840fac1c047e3fabcd74f893bc3"} Nov 03 20:52:57.038468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.038440 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" event={"ID":"51249b85-a099-4e74-a455-f539b60af19a","Type":"ContainerStarted","Data":"961aaa996c1db492b417acead4aca36ce57141d5f9d114426e6d4b021f0ee651"} Nov 03 20:52:57.040864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.040139 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" event={"ID":"97a7b8d3-b8db-4aed-853e-4e05ab491cf8","Type":"ContainerDied","Data":"19dfe3f9151ee18f10fd2ece32152cc7d953dc9c085d4d68469fce28d2b504b6"} Nov 03 20:52:57.040864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.040186 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19dfe3f9151ee18f10fd2ece32152cc7d953dc9c085d4d68469fce28d2b504b6" Nov 03 20:52:57.040864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.040259 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb" Nov 03 20:52:57.050386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.049876 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" event={"ID":"21be1d12-e5f0-4161-a549-6b3a7e9ced2b","Type":"ContainerStarted","Data":"0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6"} Nov 03 20:52:57.050386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.049916 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" event={"ID":"21be1d12-e5f0-4161-a549-6b3a7e9ced2b","Type":"ContainerStarted","Data":"5fa9e07a2bf7e9f82c3dc2e4aaa36796d4e15f885249836b6c2f01cf0205ff36"} Nov 03 20:52:57.313209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.313158 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podStartSLOduration=2.3131372150000002 podStartE2EDuration="2.313137215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:57.308149629 +0000 UTC m=+1385.373391102" watchObservedRunningTime="2026-11-03 20:52:57.313137215 +0000 UTC m=+1385.378378692" Nov 03 20:52:57.453655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.453595 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2"] Nov 03 20:52:57.554799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.553878 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:57.554799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.554029 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:57.554799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.554095 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:57.711031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.710924 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:57.712360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.712335 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:57.724964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.724937 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:52:57.907806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.907762 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:57.914428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:57.914397 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:58.077246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.077149 5932 generic.go:356] "Generic (PLEG): container finished" podID="51249b85-a099-4e74-a455-f539b60af19a" containerID="93675c229ae9fce7cb7e51f0222d2716122dff15ec18bf5c31b5fd9264a1cef7" exitCode=0 Nov 03 20:52:58.077426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.077239 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" event={"ID":"51249b85-a099-4e74-a455-f539b60af19a","Type":"ContainerDied","Data":"93675c229ae9fce7cb7e51f0222d2716122dff15ec18bf5c31b5fd9264a1cef7"} Nov 03 20:52:58.079410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.079380 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" event={"ID":"2145eea5-0bc2-4812-9538-b6325b622e38","Type":"ContainerStarted","Data":"b4daa8080a00865c328a1ec4351f965060499399fd2601731866b5347480e86a"} Nov 03 20:52:58.079527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.079430 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" event={"ID":"2145eea5-0bc2-4812-9538-b6325b622e38","Type":"ContainerStarted","Data":"b9f2d1ac71bc77582ca4c32d0892a7d10a4436659e38ab2d2038dd359e0da4f6"} Nov 03 20:52:58.080727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.079910 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:58.080801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.080783 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:52:58.108321 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.108284 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v"] Nov 03 20:52:58.109164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.109141 5932 cpu_manager.go:414] "containerMap: removing container" podUID="97a7b8d3-b8db-4aed-853e-4e05ab491cf8" containerName="revoke-keys" Nov 03 20:52:58.109164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.109161 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97a7b8d3-b8db-4aed-853e-4e05ab491cf8" containerName="revoke-keys" Nov 03 20:52:58.109351 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.109270 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="97a7b8d3-b8db-4aed-853e-4e05ab491cf8" containerName="revoke-keys" Nov 03 20:52:58.113381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.113354 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.175159 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.175114 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v"] Nov 03 20:52:58.193117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193081 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.193298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193157 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.193298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193184 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.193433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193355 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.193493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193440 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdl6c\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.193781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193752 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.193924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.193832 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.195491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.195461 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.195700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.195553 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.281523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.281495 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:58.297212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297178 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297244 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl6c\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297349 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297388 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297418 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297451 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297526 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297579 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.297655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297606 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.298001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.297944 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.298100 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.298056 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.298100 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.298084 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.298222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.298150 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.298222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.298182 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.299964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.299939 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.300150 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.300125 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.356221 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.356147 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdl6c\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.357213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.357192 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token\") pod \"e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.399110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.399069 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn\" (UniqueName: \"kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn\") pod \"2091b247-7369-41a1-9559-3ce1defa7538\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " Nov 03 20:52:58.399298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.399276 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca\") pod \"2091b247-7369-41a1-9559-3ce1defa7538\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " Nov 03 20:52:58.399421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.399400 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca\") pod \"2091b247-7369-41a1-9559-3ce1defa7538\" (UID: \"2091b247-7369-41a1-9559-3ce1defa7538\") " Nov 03 20:52:58.399760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.399734 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca" pod "2091b247-7369-41a1-9559-3ce1defa7538" (UID: "2091b247-7369-41a1-9559-3ce1defa7538"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:58.400006 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.399987 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:58.401547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.401524 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca" pod "2091b247-7369-41a1-9559-3ce1defa7538" (UID: "2091b247-7369-41a1-9559-3ce1defa7538"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:58.401707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.401687 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn" pod "2091b247-7369-41a1-9559-3ce1defa7538" (UID: "2091b247-7369-41a1-9559-3ce1defa7538"). InnerVolumeSpecName "kube-api-access-p2smn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:58.420433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.420382 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podStartSLOduration=2.4203673 podStartE2EDuration="2.4203673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:58.41671711 +0000 UTC m=+1386.481958586" watchObservedRunningTime="2026-11-03 20:52:58.4203673 +0000 UTC m=+1386.485608774" Nov 03 20:52:58.430275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.430245 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:58.501768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.501735 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2091b247-7369-41a1-9559-3ce1defa7538-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:58.501903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.501769 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2smn\" (UniqueName: \"kubernetes.io/projected/2091b247-7369-41a1-9559-3ce1defa7538-kube-api-access-p2smn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:58.823015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.822963 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v"] Nov 03 20:52:58.827604 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:52:58.827567 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ffa05d_2883_4928_889d_b17e1350ed4b.slice/crio-36cf48c351cd3060b6a25228511c9acc150920c2ef11f6a036badf0a2456cb93 WatchSource:0}: Error finding container 36cf48c351cd3060b6a25228511c9acc150920c2ef11f6a036badf0a2456cb93: Status 404 returned error can't find the container with id 36cf48c351cd3060b6a25228511c9acc150920c2ef11f6a036badf0a2456cb93 Nov 03 20:52:58.830766 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.830732 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:58.830846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.830800 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:58.830846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.830836 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:52:58.994269 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.994234 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:59.005662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:58.999895 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:59.090076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.089997 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" Nov 03 20:52:59.090076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.090022 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn" event={"ID":"2091b247-7369-41a1-9559-3ce1defa7538","Type":"ContainerDied","Data":"dc9fc41ca9d4167be097de2f4669d507ea2934c6925fea75880321ef0286a7c8"} Nov 03 20:52:59.090076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.090073 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc9fc41ca9d4167be097de2f4669d507ea2934c6925fea75880321ef0286a7c8" Nov 03 20:52:59.092217 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.092192 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" event={"ID":"76ffa05d-2883-4928-889d-b17e1350ed4b","Type":"ContainerStarted","Data":"8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b"} Nov 03 20:52:59.092322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.092239 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" event={"ID":"76ffa05d-2883-4928-889d-b17e1350ed4b","Type":"ContainerStarted","Data":"36cf48c351cd3060b6a25228511c9acc150920c2ef11f6a036badf0a2456cb93"} Nov 03 20:52:59.093018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.092998 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:59.094716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.094694 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:52:59.295850 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.295822 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:52:59.345115 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.344994 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podStartSLOduration=1.344972599 podStartE2EDuration="1.344972599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:59.341062232 +0000 UTC m=+1387.406303720" watchObservedRunningTime="2026-11-03 20:52:59.344972599 +0000 UTC m=+1387.410214077" Nov 03 20:52:59.435841 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.435783 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:52:59.447071 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.447035 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:52:59.447281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.447095 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:52:59.457132 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.457098 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca\") pod \"51249b85-a099-4e74-a455-f539b60af19a\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " Nov 03 20:52:59.457523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.457498 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca\") pod \"51249b85-a099-4e74-a455-f539b60af19a\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " Nov 03 20:52:59.457938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.457918 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl\" (UniqueName: \"kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl\") pod \"51249b85-a099-4e74-a455-f539b60af19a\" (UID: \"51249b85-a099-4e74-a455-f539b60af19a\") " Nov 03 20:52:59.459346 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.459322 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca" pod "51249b85-a099-4e74-a455-f539b60af19a" (UID: "51249b85-a099-4e74-a455-f539b60af19a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:59.463410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.463388 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca" pod "51249b85-a099-4e74-a455-f539b60af19a" (UID: "51249b85-a099-4e74-a455-f539b60af19a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:59.469876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.469854 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl" pod "51249b85-a099-4e74-a455-f539b60af19a" (UID: "51249b85-a099-4e74-a455-f539b60af19a"). InnerVolumeSpecName "kube-api-access-9vngl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:59.560171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.560139 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vngl\" (UniqueName: \"kubernetes.io/projected/51249b85-a099-4e74-a455-f539b60af19a-kube-api-access-9vngl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.560171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.560170 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51249b85-a099-4e74-a455-f539b60af19a-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:59.560414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:52:59.560183 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51249b85-a099-4e74-a455-f539b60af19a-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:00.104037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:00.104007 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" Nov 03 20:53:00.104037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:00.104029 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9" event={"ID":"51249b85-a099-4e74-a455-f539b60af19a","Type":"ContainerDied","Data":"961aaa996c1db492b417acead4aca36ce57141d5f9d114426e6d4b021f0ee651"} Nov 03 20:53:00.104515 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:00.104094 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="961aaa996c1db492b417acead4aca36ce57141d5f9d114426e6d4b021f0ee651" Nov 03 20:53:00.441088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:00.441052 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:53:00.441410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:00.441371 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:53:01.437898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.437862 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:53:01.438466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.438434 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:53:01.998533 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.998495 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh"] Nov 03 20:53:01.999608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.999584 5932 cpu_manager.go:414] "containerMap: removing container" podUID="51249b85-a099-4e74-a455-f539b60af19a" containerName="revoke-keys" Nov 03 20:53:01.999608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.999609 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51249b85-a099-4e74-a455-f539b60af19a" containerName="revoke-keys" Nov 03 20:53:01.999795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.999690 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2091b247-7369-41a1-9559-3ce1defa7538" containerName="revoke-keys" Nov 03 20:53:01.999795 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.999699 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2091b247-7369-41a1-9559-3ce1defa7538" containerName="revoke-keys" Nov 03 20:53:01.999894 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.999882 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2091b247-7369-41a1-9559-3ce1defa7538" containerName="revoke-keys" Nov 03 20:53:01.999941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:01.999905 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="51249b85-a099-4e74-a455-f539b60af19a" containerName="revoke-keys" Nov 03 20:53:02.004113 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.004092 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.087354 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.087322 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f9337cba-openshift-default-dockercfg-f46mr\"" Nov 03 20:53:02.095312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095274 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095325 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095470 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095588 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095736 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095695 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095786 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095900 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9z4z\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.095962 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095945 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.096079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.095975 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.111516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.111483 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh"] Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.209964 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210029 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210108 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210246 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9z4z\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210298 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210337 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210492 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.236666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.210530 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.256712 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.248578 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.291651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.266985 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.291651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.267537 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.291651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.277448 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.291651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.278198 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.303182 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.303143 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.431789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.431350 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:53:02.431789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.431412 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:53:02.517575 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.517496 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.659928 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.659870 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9z4z\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.663886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.663847 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.667501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.667425 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo\") pod \"e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:02.919855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:02.919820 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:03.389333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:03.389294 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh"] Nov 03 20:53:03.420247 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:03.420217 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917665c3_87ad_4ad0_b662_8358ea49e3a1.slice/crio-32ffec728fadf2769cc4339cc14721431de05fa29d88a44d24b00af6f06b4241 WatchSource:0}: Error finding container 32ffec728fadf2769cc4339cc14721431de05fa29d88a44d24b00af6f06b4241: Status 404 returned error can't find the container with id 32ffec728fadf2769cc4339cc14721431de05fa29d88a44d24b00af6f06b4241 Nov 03 20:53:03.448570 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:03.448536 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:53:03.448855 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:03.448819 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:53:03.451252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:03.451210 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:03.451381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:03.451309 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:03.451381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:03.451355 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:04.018645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.007759 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng"] Nov 03 20:53:04.241705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.163759 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.300683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.293381 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-b6eaffa8\"" Nov 03 20:53:04.300683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.293516 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-b6eaffa8-dockercfg-j7b5t\"" Nov 03 20:53:04.322390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.322336 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.322941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.322795 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgpjb\" (UniqueName: \"kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb\") pod \"payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.377148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.377110 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng"] Nov 03 20:53:04.398789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.398760 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" event={"ID":"917665c3-87ad-4ad0-b662-8358ea49e3a1","Type":"ContainerStarted","Data":"ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8"} Nov 03 20:53:04.399033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.399012 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" event={"ID":"917665c3-87ad-4ad0-b662-8358ea49e3a1","Type":"ContainerStarted","Data":"32ffec728fadf2769cc4339cc14721431de05fa29d88a44d24b00af6f06b4241"} Nov 03 20:53:04.438653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.433541 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.438653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.433671 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgpjb\" (UniqueName: \"kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb\") pod \"payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.452649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.446978 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:53:04.452649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.447042 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:53:04.453025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.452986 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.491685 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.491653 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgpjb\" (UniqueName: \"kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb\") pod \"payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.518881 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.518786 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podStartSLOduration=3.518772384 podStartE2EDuration="3.518772384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:04.516466459 +0000 UTC m=+1392.581707931" watchObservedRunningTime="2026-11-03 20:53:04.518772384 +0000 UTC m=+1392.584013938" Nov 03 20:53:04.579605 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.579573 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:04.930645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.921044 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:04.957654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:04.957600 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:05.149461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.149430 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng"] Nov 03 20:53:05.151616 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:05.151585 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5dc2c2_a7ca_46d4_b33c_c368cdc6375e.slice/crio-89a88e3645933504622c7a92965260d9cebcee219bd692ac56b6ee9166a6466a WatchSource:0}: Error finding container 89a88e3645933504622c7a92965260d9cebcee219bd692ac56b6ee9166a6466a: Status 404 returned error can't find the container with id 89a88e3645933504622c7a92965260d9cebcee219bd692ac56b6ee9166a6466a Nov 03 20:53:05.314589 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.314547 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx"] Nov 03 20:53:05.319287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.319268 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.379655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.377814 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:53:05.379655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.377910 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:05.388259 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.388230 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx"] Nov 03 20:53:05.416094 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.415997 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" event={"ID":"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e","Type":"ContainerStarted","Data":"68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57"} Nov 03 20:53:05.416094 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.416071 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" event={"ID":"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e","Type":"ContainerStarted","Data":"89a88e3645933504622c7a92965260d9cebcee219bd692ac56b6ee9166a6466a"} Nov 03 20:53:05.416358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.416105 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:05.416539 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.416516 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:05.417614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.417593 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:53:05.431053 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.431023 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 20:53:05.431185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.431071 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 20:53:05.464017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.463983 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wk92\" (UniqueName: \"kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.464191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.464174 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.464257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.464237 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.570870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.569681 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.570870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.569913 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wk92\" (UniqueName: \"kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.570870 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.570161 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.571203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.571047 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.609693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.604758 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.632467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.632431 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wk92\" (UniqueName: \"kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92\") pod \"maas-api-revoke-sub-d5d2cb38-j5mmx\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.638341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.638307 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:05.667661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:05.667496 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" podStartSLOduration=2.667474463 podStartE2EDuration="2.667474463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:05.667152942 +0000 UTC m=+1393.732394416" watchObservedRunningTime="2026-11-03 20:53:05.667474463 +0000 UTC m=+1393.732715940" Nov 03 20:53:06.030670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.030551 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b"] Nov 03 20:53:06.036610 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.036579 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.049433 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:06.049385 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15be16f_9973_48bb_ae38_618c04442d61.slice/crio-4d54d0d005aa5275af9b5458d931b3d82f5e8ecc97073080ddd3f0c7378bd28f WatchSource:0}: Error finding container 4d54d0d005aa5275af9b5458d931b3d82f5e8ecc97073080ddd3f0c7378bd28f: Status 404 returned error can't find the container with id 4d54d0d005aa5275af9b5458d931b3d82f5e8ecc97073080ddd3f0c7378bd28f Nov 03 20:53:06.056682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.056659 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx"] Nov 03 20:53:06.088891 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.088446 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b"] Nov 03 20:53:06.185257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.185212 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkvnm\" (UniqueName: \"kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.185800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.185385 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.185800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.185591 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.287760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.287659 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkvnm\" (UniqueName: \"kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.287941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.287779 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.287941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.287839 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.288479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.288449 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.323053 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.323007 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.345313 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.345277 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkvnm\" (UniqueName: \"kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm\") pod \"maas-api-revoke-sub-35e8162b-lsw6b\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.371450 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.371420 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:06.432837 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.432799 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" event={"ID":"d15be16f-9973-48bb-ae38-618c04442d61","Type":"ContainerStarted","Data":"4d54d0d005aa5275af9b5458d931b3d82f5e8ecc97073080ddd3f0c7378bd28f"} Nov 03 20:53:06.777239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.777204 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b"] Nov 03 20:53:06.779780 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:06.779747 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7147335_1917_4975_b8a1_f66f39b0bf7f.slice/crio-478ee29fe1dab61b35957c56f3bbde0882c7bd932d51ce0d3f60c4936ba731ec WatchSource:0}: Error finding container 478ee29fe1dab61b35957c56f3bbde0882c7bd932d51ce0d3f60c4936ba731ec: Status 404 returned error can't find the container with id 478ee29fe1dab61b35957c56f3bbde0882c7bd932d51ce0d3f60c4936ba731ec Nov 03 20:53:06.946174 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.946139 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:53:06.947407 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.947384 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:53:06.948435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:06.948409 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 20:53:07.456357 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:07.456314 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" event={"ID":"c7147335-1917-4975-b8a1-f66f39b0bf7f","Type":"ContainerStarted","Data":"e923ef7d29d869b36a6fa382871d9e1aacbbfabd6b3a80f2697f99b108bcd069"} Nov 03 20:53:07.456981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:07.456950 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" event={"ID":"c7147335-1917-4975-b8a1-f66f39b0bf7f","Type":"ContainerStarted","Data":"478ee29fe1dab61b35957c56f3bbde0882c7bd932d51ce0d3f60c4936ba731ec"} Nov 03 20:53:07.458700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:07.458674 5932 generic.go:356] "Generic (PLEG): container finished" podID="d15be16f-9973-48bb-ae38-618c04442d61" containerID="554a1c43d87c2501dc0007e2259816c9b68cdd8ae1f7e1f0c62b728bba99854c" exitCode=0 Nov 03 20:53:07.458802 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:07.458747 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" event={"ID":"d15be16f-9973-48bb-ae38-618c04442d61","Type":"ContainerDied","Data":"554a1c43d87c2501dc0007e2259816c9b68cdd8ae1f7e1f0c62b728bba99854c"} Nov 03 20:53:07.565557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:07.565487 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" podStartSLOduration=2.565468502 podStartE2EDuration="2.565468502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:07.564670661 +0000 UTC m=+1395.629912131" watchObservedRunningTime="2026-11-03 20:53:07.565468502 +0000 UTC m=+1395.630709977" Nov 03 20:53:08.469664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.469601 5932 generic.go:356] "Generic (PLEG): container finished" podID="c7147335-1917-4975-b8a1-f66f39b0bf7f" containerID="e923ef7d29d869b36a6fa382871d9e1aacbbfabd6b3a80f2697f99b108bcd069" exitCode=0 Nov 03 20:53:08.470039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.469687 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" event={"ID":"c7147335-1917-4975-b8a1-f66f39b0bf7f","Type":"ContainerDied","Data":"e923ef7d29d869b36a6fa382871d9e1aacbbfabd6b3a80f2697f99b108bcd069"} Nov 03 20:53:08.643435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.643406 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:08.729776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.729688 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca\") pod \"d15be16f-9973-48bb-ae38-618c04442d61\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " Nov 03 20:53:08.729776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.729727 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92\" (UniqueName: \"kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92\") pod \"d15be16f-9973-48bb-ae38-618c04442d61\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " Nov 03 20:53:08.729776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.729763 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca\") pod \"d15be16f-9973-48bb-ae38-618c04442d61\" (UID: \"d15be16f-9973-48bb-ae38-618c04442d61\") " Nov 03 20:53:08.730134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.730107 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca" pod "d15be16f-9973-48bb-ae38-618c04442d61" (UID: "d15be16f-9973-48bb-ae38-618c04442d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:08.730392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.730373 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:08.731954 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.731927 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92" pod "d15be16f-9973-48bb-ae38-618c04442d61" (UID: "d15be16f-9973-48bb-ae38-618c04442d61"). InnerVolumeSpecName "kube-api-access-7wk92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:08.732079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.732058 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca" pod "d15be16f-9973-48bb-ae38-618c04442d61" (UID: "d15be16f-9973-48bb-ae38-618c04442d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:08.836653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.833480 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wk92\" (UniqueName: \"kubernetes.io/projected/d15be16f-9973-48bb-ae38-618c04442d61-kube-api-access-7wk92\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:08.836653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:08.833521 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d15be16f-9973-48bb-ae38-618c04442d61-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:09.496243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.495733 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" Nov 03 20:53:09.496243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.495788 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx" event={"ID":"d15be16f-9973-48bb-ae38-618c04442d61","Type":"ContainerDied","Data":"4d54d0d005aa5275af9b5458d931b3d82f5e8ecc97073080ddd3f0c7378bd28f"} Nov 03 20:53:09.496243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.495826 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d54d0d005aa5275af9b5458d931b3d82f5e8ecc97073080ddd3f0c7378bd28f" Nov 03 20:53:09.663924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.663895 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:09.727492 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.727461 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca\") pod \"c7147335-1917-4975-b8a1-f66f39b0bf7f\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " Nov 03 20:53:09.727714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.727591 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca\") pod \"c7147335-1917-4975-b8a1-f66f39b0bf7f\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " Nov 03 20:53:09.727789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.727739 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm\" (UniqueName: \"kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm\") pod \"c7147335-1917-4975-b8a1-f66f39b0bf7f\" (UID: \"c7147335-1917-4975-b8a1-f66f39b0bf7f\") " Nov 03 20:53:09.727888 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.727851 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca" pod "c7147335-1917-4975-b8a1-f66f39b0bf7f" (UID: "c7147335-1917-4975-b8a1-f66f39b0bf7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:09.728243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.728225 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:09.729716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.729694 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca" pod "c7147335-1917-4975-b8a1-f66f39b0bf7f" (UID: "c7147335-1917-4975-b8a1-f66f39b0bf7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:09.729789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.729727 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm" pod "c7147335-1917-4975-b8a1-f66f39b0bf7f" (UID: "c7147335-1917-4975-b8a1-f66f39b0bf7f"). InnerVolumeSpecName "kube-api-access-qkvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:09.829873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.829797 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7147335-1917-4975-b8a1-f66f39b0bf7f-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:09.829873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:09.829823 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkvnm\" (UniqueName: \"kubernetes.io/projected/c7147335-1917-4975-b8a1-f66f39b0bf7f-kube-api-access-qkvnm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:10.505433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:10.505404 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" Nov 03 20:53:10.505836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:10.505392 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b" event={"ID":"c7147335-1917-4975-b8a1-f66f39b0bf7f","Type":"ContainerDied","Data":"478ee29fe1dab61b35957c56f3bbde0882c7bd932d51ce0d3f60c4936ba731ec"} Nov 03 20:53:10.505836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:10.505532 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="478ee29fe1dab61b35957c56f3bbde0882c7bd932d51ce0d3f60c4936ba731ec" Nov 03 20:53:11.590274 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.590237 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf"] Nov 03 20:53:11.591166 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.591146 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c7147335-1917-4975-b8a1-f66f39b0bf7f" containerName="revoke-keys" Nov 03 20:53:11.591229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.591168 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7147335-1917-4975-b8a1-f66f39b0bf7f" containerName="revoke-keys" Nov 03 20:53:11.591229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.591181 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d15be16f-9973-48bb-ae38-618c04442d61" containerName="revoke-keys" Nov 03 20:53:11.591229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.591186 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15be16f-9973-48bb-ae38-618c04442d61" containerName="revoke-keys" Nov 03 20:53:11.591332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.591284 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7147335-1917-4975-b8a1-f66f39b0bf7f" containerName="revoke-keys" Nov 03 20:53:11.591332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.591294 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15be16f-9973-48bb-ae38-618c04442d61" containerName="revoke-keys" Nov 03 20:53:11.595009 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.594992 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.656824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.656787 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.656899 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.656939 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.656993 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.657025 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd6zp\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.657059 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.657073 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.657126 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.657212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.657183 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.698838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.698798 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf"] Nov 03 20:53:11.758334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758301 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758341 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758366 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758400 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758665 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758791 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758871 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758856 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758861 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.758929 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758885 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.759032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758944 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd6zp\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.759102 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.758796 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.759190 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.759138 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.759463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.759439 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.759725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.759700 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.760944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.760915 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.761311 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.761293 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.816885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.816840 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd6zp\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.817069 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.816923 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:11.909432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:11.909401 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:12.353882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.353801 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf"] Nov 03 20:53:12.391582 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:12.391552 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0782bf0_b851_4167_ab65_38ba0be007d2.slice/crio-0d0409efd4e70dcb26d6c60264aebb5dd60eca5b4803fa6ee606238d241a9991 WatchSource:0}: Error finding container 0d0409efd4e70dcb26d6c60264aebb5dd60eca5b4803fa6ee606238d241a9991: Status 404 returned error can't find the container with id 0d0409efd4e70dcb26d6c60264aebb5dd60eca5b4803fa6ee606238d241a9991 Nov 03 20:53:12.394189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.394153 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:12.394279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.394220 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:12.394279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.394270 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:12.545811 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.545778 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg"] Nov 03 20:53:12.552280 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.552258 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.559928 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.559886 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" event={"ID":"f0782bf0-b851-4167-ab65-38ba0be007d2","Type":"ContainerStarted","Data":"0d0409efd4e70dcb26d6c60264aebb5dd60eca5b4803fa6ee606238d241a9991"} Nov 03 20:53:12.647643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.643591 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg"] Nov 03 20:53:12.686012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.685974 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzv4\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.686195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686061 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.686195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686173 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.686296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686215 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.686330 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686301 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.686364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686327 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.687692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686411 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.687692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686432 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.687692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.686475 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.787918 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.787883 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.787937 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.787981 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788006 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788073 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788099 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788176 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788329 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzv4\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788350 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788424 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788436 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788736 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788836 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.788880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.788847 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.790657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.790597 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.791033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.791012 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.845593 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.845562 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.845916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.845891 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzv4\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4\") pod \"e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:12.875588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:12.875561 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:13.320225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.320044 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg"] Nov 03 20:53:13.323287 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:13.323244 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0087567b_0be5_4ce8_816c_1ce888df102b.slice/crio-56cdcbf6b2fbbe481b8dec84ec0377dfe682a1873c5c5cca50afeeda1086429f WatchSource:0}: Error finding container 56cdcbf6b2fbbe481b8dec84ec0377dfe682a1873c5c5cca50afeeda1086429f: Status 404 returned error can't find the container with id 56cdcbf6b2fbbe481b8dec84ec0377dfe682a1873c5c5cca50afeeda1086429f Nov 03 20:53:13.325941 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.325908 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:13.326005 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.325978 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:13.326045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.326005 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:13.576997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.576904 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" event={"ID":"0087567b-0be5-4ce8-816c-1ce888df102b","Type":"ContainerStarted","Data":"d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164"} Nov 03 20:53:13.577444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.577408 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" event={"ID":"0087567b-0be5-4ce8-816c-1ce888df102b","Type":"ContainerStarted","Data":"56cdcbf6b2fbbe481b8dec84ec0377dfe682a1873c5c5cca50afeeda1086429f"} Nov 03 20:53:13.579314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.579284 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" event={"ID":"f0782bf0-b851-4167-ab65-38ba0be007d2","Type":"ContainerStarted","Data":"eda28d3386368d36b72faad2b330a0cc8f53a78044ab06b2c99c77d4c9c75f41"} Nov 03 20:53:13.779025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.778953 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podStartSLOduration=1.7789277970000001 podStartE2EDuration="1.778927797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:13.729002478 +0000 UTC m=+1401.794243955" watchObservedRunningTime="2026-11-03 20:53:13.778927797 +0000 UTC m=+1401.844169270" Nov 03 20:53:13.883644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.878185 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:13.891981 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.884864 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podStartSLOduration=2.884840887 podStartE2EDuration="2.884840887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:13.872113111 +0000 UTC m=+1401.937354590" watchObservedRunningTime="2026-11-03 20:53:13.884840887 +0000 UTC m=+1401.950082362" Nov 03 20:53:13.943654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.925189 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:13.968654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.944747 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:13.968654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:13.944830 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:14.664108 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:53:14.664069 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:53:14.877964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:14.877928 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:14.878382 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:14.877986 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:14.909844 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:14.909811 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:14.910017 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:14.909882 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:15.877504 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:15.877466 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:15.877829 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:15.877795 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:15.913543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:15.913501 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:15.914144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:15.914094 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:16.484650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:16.473814 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:53:16.881553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:16.881001 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:16.881553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:16.881059 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:16.910568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:16.910522 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:16.910761 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:16.910593 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:17.876748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:17.876562 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:17.880003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:17.877736 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:17.910103 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:17.910068 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:17.910265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:17.910119 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:18.876183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:18.876144 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:18.876554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:18.876514 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:18.910411 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:18.910374 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:18.910843 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:18.910436 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:19.876026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:19.875990 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:19.876251 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:19.876054 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:19.910015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:19.909976 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:19.910229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:19.910041 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:20.876134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:20.876094 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 20:53:20.876602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:20.876153 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 20:53:20.910372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:20.910341 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body= Nov 03 20:53:20.910563 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:20.910398 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" Nov 03 20:53:22.317422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:22.317382 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:22.319315 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:22.319255 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:22.320076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:22.320048 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:53:22.501557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:22.501525 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:22.501959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:22.501891 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:22.502775 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:22.502755 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 20:53:25.568779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:25.568729 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:25.802647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:25.801719 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:25.886499 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:25.885734 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:26.249709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:26.249588 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:27.306646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.271759 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:53:27.333647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.324050 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 20:53:27.356158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.353332 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:27.356158 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.353549 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 20:53:27.419823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.365707 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:27.419823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.387993 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:53:27.517149 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:53:27.517109 5932 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.37s" Nov 03 20:53:27.517442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.517424 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:53:28.294647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.517613 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:53:28.311649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.615785 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-qr779" podUID="b476286d-e752-4bdc-9441-5518c4860345" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Nov 03 20:53:28.311649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.616011 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-5vmsj" podUID="87927855-684d-4f9a-8614-dd9649b8af0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:28.367648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:27.857690 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps" podUID="3ab4bbe3-d96e-4e91-9345-d76342e8dc9b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:28.381648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.336434 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:28.381648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.353858 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 20:53:28.381648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.353893 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:53:28.381648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.353922 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:28.387288 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.353953 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:28.387799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.367741 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:53:28.406653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.397919 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:28.473146 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:28.473087 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:29.014025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.013990 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:53:29.014323 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.014305 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:53:29.014453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.014438 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq"] Nov 03 20:53:29.084015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.083400 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:53:29.084015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.083469 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:53:29.084015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.083600 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq"] Nov 03 20:53:29.084655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.084610 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.157729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.143280 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.314044 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.313953 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmxh\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.316149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.316120 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.319110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.319081 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.319335 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.319315 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.319753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.319732 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.319887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.319871 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.320422 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.320402 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.320580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.320565 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.320865 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.298267 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r"] Nov 03 20:53:29.401911 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.401867 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:53:29.454281 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.449287 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.463646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.455689 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.464106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.464070 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.482176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.482142 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.494405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.494369 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.494754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.494724 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmxh\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.519747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.519713 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.520064 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.520039 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.520207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.520190 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.520479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.481930 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:53:29.520737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.481951 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 20:53:29.522252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.481806 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:53:29.522252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.481877 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:53:29.522252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.521941 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:29.522252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.522190 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.522513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.522329 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.522892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.522848 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.571799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.546886 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx"] Nov 03 20:53:29.611955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.611865 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.821868 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.643864 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r"] Nov 03 20:53:29.822167 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.822150 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx"] Nov 03 20:53:29.822334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.822323 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:53:29.822723 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.752809 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmxh\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.822821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.783820 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.822821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.789249 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.822821 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.643959 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.822994 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.777940 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.842561 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.842523 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:53:29.852394 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.852364 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:53:29.856876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.856851 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:53:29.867257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.867229 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:29.867537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.867506 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:29.918464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.918426 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.918773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.918749 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.918959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.918938 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.919356 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.919321 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.919501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.919483 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8j69\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.919700 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.919680 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.919825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.919809 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mwt\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.919960 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.919943 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.920089 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920071 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.920206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920189 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.920418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920400 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.920542 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920526 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.920692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920675 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.920828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920812 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.920940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.920925 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.921111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.921093 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.921309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.921288 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:29.921426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.921411 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:29.926978 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.926957 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:53:29.927161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:29.927147 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.030865 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.030912 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.030966 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031000 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031029 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031085 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031177 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031210 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031249 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031352 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031424 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031482 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031515 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8j69\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031699 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031751 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mwt\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031787 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031831 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.031998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.031865 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.033455 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.033429 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.033989 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.033940 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.034524 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.034492 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.035397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.035373 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.038473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.038448 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.042068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.039422 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.042068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.039720 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.042068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.039999 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.042068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.041398 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.045185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.045163 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.046196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.046174 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.046602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.046569 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.056286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.056260 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.087691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.084825 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo\") pod \"e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:30.155940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.155904 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8j69\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.171936 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.159136 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mwt\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.274880 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.274845 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.319121 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.318828 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.322185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.319142 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:30.328411 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.327923 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token\") pod \"e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.356461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.356383 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:30.388384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.388093 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:30.910106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.910074 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c"] Nov 03 20:53:30.933386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.933356 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c"] Nov 03 20:53:30.933666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.933650 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57"] Nov 03 20:53:30.957357 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.947219 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:30.967586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.964128 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57"] Nov 03 20:53:30.967586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.964289 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.010898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:30.999680 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d9a2c67d-dockercfg-xmd8c\"" Nov 03 20:53:31.010898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.000157 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d9a2c67d\"" Nov 03 20:53:31.089109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.089027 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8tg\" (UniqueName: \"kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg\") pod \"payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.089255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.089198 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mds2\" (UniqueName: \"kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2\") pod \"payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.089326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.089299 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.089419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.089410 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.195716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.191453 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.195716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.191607 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.195716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.191888 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8tg\" (UniqueName: \"kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg\") pod \"payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.195716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.191960 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mds2\" (UniqueName: \"kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2\") pod \"payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.195716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.193178 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.195716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.194762 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume\") pod \"payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.273943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.273888 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8tg\" (UniqueName: \"kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg\") pod \"payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.274087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.273959 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mds2\" (UniqueName: \"kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2\") pod \"payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.288944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.288904 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.307498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.307462 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:31.377185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.376986 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r"] Nov 03 20:53:31.422074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.422045 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq"] Nov 03 20:53:31.426144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.425191 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.426144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.425294 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.426144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.425335 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.438373 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.438336 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.438483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.438419 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.438483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.438463 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.535876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.535784 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx"] Nov 03 20:53:31.545057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.545023 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.545149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.545092 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.545149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.545123 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:31.766045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.762914 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c"] Nov 03 20:53:31.769511 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:31.769460 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b250057_e693_476f_b75f_afb439b384a8.slice/crio-ccb18b931c3c1f545ecaf1818e9e7b25486c99fd4a296b17275a5dd0209cbc6d WatchSource:0}: Error finding container ccb18b931c3c1f545ecaf1818e9e7b25486c99fd4a296b17275a5dd0209cbc6d: Status 404 returned error can't find the container with id ccb18b931c3c1f545ecaf1818e9e7b25486c99fd4a296b17275a5dd0209cbc6d Nov 03 20:53:31.876594 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.875224 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57"] Nov 03 20:53:31.904892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.904006 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" event={"ID":"5b250057-e693-476f-b75f-afb439b384a8","Type":"ContainerStarted","Data":"5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1"} Nov 03 20:53:31.904892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.904072 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" event={"ID":"5b250057-e693-476f-b75f-afb439b384a8","Type":"ContainerStarted","Data":"ccb18b931c3c1f545ecaf1818e9e7b25486c99fd4a296b17275a5dd0209cbc6d"} Nov 03 20:53:31.904892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.904352 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:31.926485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.926441 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" event={"ID":"4d3e97ff-af2c-4099-94b1-e5d45b325b2d","Type":"ContainerStarted","Data":"261f2cf849c0454a2b201608f46114caadf5b4e43858b6f0f8222140b4cfe0d9"} Nov 03 20:53:31.926792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.926773 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" event={"ID":"4d3e97ff-af2c-4099-94b1-e5d45b325b2d","Type":"ContainerStarted","Data":"6899f504250df9b95676cc1fe632b543775c9ff26d9ac5d821e955d6e11f874e"} Nov 03 20:53:31.929397 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.929373 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" event={"ID":"c2d88b32-d6e3-4077-a805-2d88867c93ad","Type":"ContainerStarted","Data":"f95d30db4e42440be84837d49ddf5fddb207ebdec337b1d1e2a043bca952698b"} Nov 03 20:53:31.929570 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.929556 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" event={"ID":"c2d88b32-d6e3-4077-a805-2d88867c93ad","Type":"ContainerStarted","Data":"f82bbb008ed0af793883d83a8b1f8a2b738f353ea6c062e69acafb4657fe2392"} Nov 03 20:53:31.937446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.936919 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" event={"ID":"4297f2be-be32-4a9a-bd71-c1d294e1f569","Type":"ContainerStarted","Data":"974401bb5ad969553fc0e1b40a98f4aa6e0f22edca768219849baa9fe4784c49"} Nov 03 20:53:31.937446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.936971 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" event={"ID":"4297f2be-be32-4a9a-bd71-c1d294e1f569","Type":"ContainerStarted","Data":"83c05b019c28f97a805583453f341f28202138af19f5dae5354109e7e82df9bc"} Nov 03 20:53:31.940553 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:31.940528 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" event={"ID":"da90ae73-1008-4edd-9b3e-c4011c77a46f","Type":"ContainerStarted","Data":"d7b69a0e5777da762e06d6f8a2263792c03bf0ce55418c1b59d09e099b46c9c6"} Nov 03 20:53:32.014237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.014172 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" podStartSLOduration=2.014153073 podStartE2EDuration="2.014153073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:32.012442766 +0000 UTC m=+1420.077684239" watchObservedRunningTime="2026-11-03 20:53:32.014153073 +0000 UTC m=+1420.079394547" Nov 03 20:53:32.195147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.195079 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podStartSLOduration=5.195053652 podStartE2EDuration="5.195053652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:32.135060573 +0000 UTC m=+1420.200302047" watchObservedRunningTime="2026-11-03 20:53:32.195053652 +0000 UTC m=+1420.260295152" Nov 03 20:53:32.280703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.276270 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podStartSLOduration=5.276247705 podStartE2EDuration="5.276247705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:32.274029888 +0000 UTC m=+1420.339271465" watchObservedRunningTime="2026-11-03 20:53:32.276247705 +0000 UTC m=+1420.341489180" Nov 03 20:53:32.324508 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.324476 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:32.325968 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.325930 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:32.326131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.325988 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:32.369171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.361693 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:32.369171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.363730 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:32.369171 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.363800 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:32.416945 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.416902 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:32.417499 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.417051 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:32.417499 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.417110 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:32.424671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:32.424232 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podStartSLOduration=6.424210314 podStartE2EDuration="6.424210314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:32.417525611 +0000 UTC m=+1420.482767084" watchObservedRunningTime="2026-11-03 20:53:32.424210314 +0000 UTC m=+1420.489451790" Nov 03 20:53:33.011290 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.011248 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" event={"ID":"da90ae73-1008-4edd-9b3e-c4011c77a46f","Type":"ContainerStarted","Data":"77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb"} Nov 03 20:53:33.012024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.011999 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:33.336737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.333047 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:33.336737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.333111 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:33.365737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.363091 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:33.365737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.363163 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:33.406368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.406243 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:33.406368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:33.406311 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:34.326656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:34.325213 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:34.326656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:34.325290 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:34.363650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:34.362731 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:34.363650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:34.362885 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:34.400555 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:34.400521 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:34.400770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:34.400572 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:35.325123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:35.325078 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:35.325445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:35.325413 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:35.360282 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:35.360244 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:35.360924 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:35.360891 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:35.402474 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:35.401034 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:35.402474 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:35.401116 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:36.342650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:36.333205 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:36.342650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:36.333288 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:36.365648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:36.360071 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:36.365648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:36.360156 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:36.405646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:36.404137 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:36.405646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:36.404189 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:37.360653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:37.336301 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:37.360653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:37.336388 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:37.364646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:37.361437 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:37.364646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:37.361511 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:37.407646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:37.406751 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:37.407646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:37.406814 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:38.325673 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:38.325581 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body= Nov 03 20:53:38.325673 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:38.325667 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" Nov 03 20:53:38.359395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:38.359354 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 03 20:53:38.359573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:38.359411 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 03 20:53:38.400298 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:38.400255 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 03 20:53:38.400512 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:38.400324 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 03 20:53:39.492793 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.492724 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" podStartSLOduration=9.492702268 podStartE2EDuration="9.492702268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:33.144677397 +0000 UTC m=+1421.209918871" watchObservedRunningTime="2026-11-03 20:53:39.492702268 +0000 UTC m=+1427.557943742" Nov 03 20:53:39.493532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.493504 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w"] Nov 03 20:53:39.501502 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.501475 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.547648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.544618 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk5v9\" (UniqueName: \"kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.547648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.544739 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.547648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.544853 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.556749 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.556711 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w"] Nov 03 20:53:39.636348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.636312 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:53:39.637310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.637283 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:53:39.655668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.655605 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk5v9\" (UniqueName: \"kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.655940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.655916 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.656147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.656127 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.657318 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.657295 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.661050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.661030 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.700413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.700374 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:39.701977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.701935 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:39.705925 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.705890 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:53:39.717642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.717598 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk5v9\" (UniqueName: \"kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9\") pod \"maas-api-revoke-sub-0f727b7f-wpl4w\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:39.772571 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.772485 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:39.773761 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.773705 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:39.774883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.774859 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:53:39.784871 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.784842 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:39.785208 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.785185 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:39.786065 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.786036 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:53:39.832230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:39.832196 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:40.228263 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:40.228234 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w"] Nov 03 20:53:40.231289 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:40.231258 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b195c12_c233_4638_983b_383700b4c6bb.slice/crio-010ba957ada5ff873a0b0a0c4991b4abe968a34676a64281208a11ad69b7cd9d WatchSource:0}: Error finding container 010ba957ada5ff873a0b0a0c4991b4abe968a34676a64281208a11ad69b7cd9d: Status 404 returned error can't find the container with id 010ba957ada5ff873a0b0a0c4991b4abe968a34676a64281208a11ad69b7cd9d Nov 03 20:53:41.207561 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:41.207526 5932 generic.go:356] "Generic (PLEG): container finished" podID="0b195c12-c233-4638-983b-383700b4c6bb" containerID="e8c405912afdb0099b1c030a84a4d602399eb8eabc49ea7835ea811bb7bc1f65" exitCode=0 Nov 03 20:53:41.207952 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:41.207612 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" event={"ID":"0b195c12-c233-4638-983b-383700b4c6bb","Type":"ContainerDied","Data":"e8c405912afdb0099b1c030a84a4d602399eb8eabc49ea7835ea811bb7bc1f65"} Nov 03 20:53:41.207952 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:41.207681 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" event={"ID":"0b195c12-c233-4638-983b-383700b4c6bb","Type":"ContainerStarted","Data":"010ba957ada5ff873a0b0a0c4991b4abe968a34676a64281208a11ad69b7cd9d"} Nov 03 20:53:42.410371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.410338 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:42.523250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.523074 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca\") pod \"0b195c12-c233-4638-983b-383700b4c6bb\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " Nov 03 20:53:42.523250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.523154 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9\" (UniqueName: \"kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9\") pod \"0b195c12-c233-4638-983b-383700b4c6bb\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " Nov 03 20:53:42.523471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.523278 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca\") pod \"0b195c12-c233-4638-983b-383700b4c6bb\" (UID: \"0b195c12-c233-4638-983b-383700b4c6bb\") " Nov 03 20:53:42.523882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.523828 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca" pod "0b195c12-c233-4638-983b-383700b4c6bb" (UID: "0b195c12-c233-4638-983b-383700b4c6bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:53:42.524187 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.524138 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:42.526449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.526419 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca" pod "0b195c12-c233-4638-983b-383700b4c6bb" (UID: "0b195c12-c233-4638-983b-383700b4c6bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:42.526906 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.526876 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9" pod "0b195c12-c233-4638-983b-383700b4c6bb" (UID: "0b195c12-c233-4638-983b-383700b4c6bb"). InnerVolumeSpecName "kube-api-access-fk5v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:42.625786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.625747 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk5v9\" (UniqueName: \"kubernetes.io/projected/0b195c12-c233-4638-983b-383700b4c6bb-kube-api-access-fk5v9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:42.625786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:42.625782 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b195c12-c233-4638-983b-383700b4c6bb-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:43.015545 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:43.015511 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:53:43.240572 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:43.240532 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" event={"ID":"0b195c12-c233-4638-983b-383700b4c6bb","Type":"ContainerDied","Data":"010ba957ada5ff873a0b0a0c4991b4abe968a34676a64281208a11ad69b7cd9d"} Nov 03 20:53:43.240760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:43.240595 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w" Nov 03 20:53:43.240760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:43.240612 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="010ba957ada5ff873a0b0a0c4991b4abe968a34676a64281208a11ad69b7cd9d" Nov 03 20:53:44.085163 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:44.085130 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:53:44.153525 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:53:44.153480 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:53:46.138395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.138359 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r"] Nov 03 20:53:46.139224 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.139207 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0b195c12-c233-4638-983b-383700b4c6bb" containerName="revoke-keys" Nov 03 20:53:46.139265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.139225 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b195c12-c233-4638-983b-383700b4c6bb" containerName="revoke-keys" Nov 03 20:53:46.139408 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.139397 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b195c12-c233-4638-983b-383700b4c6bb" containerName="revoke-keys" Nov 03 20:53:46.144396 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.144376 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.290448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290409 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.290695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290673 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.290769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290711 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.290769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290752 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.290899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290815 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.290965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290953 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8qj\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.291001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.290988 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.291056 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.291022 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.291099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.291076 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.347105 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.347063 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r"] Nov 03 20:53:46.392481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392387 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392436 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392487 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392519 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392590 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8qj\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392657 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392707 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392735 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.392983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392838 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.393076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.392993 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.393222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.393198 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.393299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.393281 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.393757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.393718 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.393937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.393789 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.395663 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.395600 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.396183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.396156 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.450318 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.450281 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.450487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.450456 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8qj\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-vml9r\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.459789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.459762 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:46.923322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.923293 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r"] Nov 03 20:53:46.925846 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:53:46.925811 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbd4e68_62c9_4fde_b3f3_ca809cb6f13c.slice/crio-a9a9780ffadfc88e3b5a2f1148b246e59c12802e54557fee876771352f248ec6 WatchSource:0}: Error finding container a9a9780ffadfc88e3b5a2f1148b246e59c12802e54557fee876771352f248ec6: Status 404 returned error can't find the container with id a9a9780ffadfc88e3b5a2f1148b246e59c12802e54557fee876771352f248ec6 Nov 03 20:53:46.928858 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.928827 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:46.928959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.928899 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:46.928959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:46.928929 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:53:47.289989 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:47.287588 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" event={"ID":"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c","Type":"ContainerStarted","Data":"fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042"} Nov 03 20:53:47.291076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:47.291015 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" event={"ID":"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c","Type":"ContainerStarted","Data":"a9a9780ffadfc88e3b5a2f1148b246e59c12802e54557fee876771352f248ec6"} Nov 03 20:53:47.418220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:47.418155 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podStartSLOduration=1.4181329489999999 podStartE2EDuration="1.418132949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:53:47.416225718 +0000 UTC m=+1435.481467192" watchObservedRunningTime="2026-11-03 20:53:47.418132949 +0000 UTC m=+1435.483374423" Nov 03 20:53:47.466456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:47.466421 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:47.469727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:47.469308 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:47.469727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:47.469368 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:48.460025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:48.459990 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:48.460519 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:48.460044 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:49.472666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:49.464758 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:49.472666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:49.464837 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:50.460735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:50.460694 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:50.460956 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:50.460767 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:51.460921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:51.460884 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:51.461419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:51.460931 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:52.466645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:52.466231 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:52.466645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:52.466314 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:53.111018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.110881 5932 scope.go:122] "RemoveContainer" containerID="566fff1e82780550e6b59fec0741246f231483c885c5806548a1d245c1bad207" Nov 03 20:53:53.135118 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.131818 5932 scope.go:122] "RemoveContainer" containerID="8698ce80a33bdf0153abb2061c68c0f1da5408f5edcc287e70ba3b27a5280d7d" Nov 03 20:53:53.159480 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.159449 5932 scope.go:122] "RemoveContainer" containerID="6c9867f045d30f96ddb2f267873ebcf9c3a5d341e048bf02c037463b912ca75d" Nov 03 20:53:53.171852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.171781 5932 scope.go:122] "RemoveContainer" containerID="d1e6eb81fccb61319a65f65acd465dd32806b174a02e9e399f9cd06469b522da" Nov 03 20:53:53.185571 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.185549 5932 scope.go:122] "RemoveContainer" containerID="777ca4f807ebd18ba9b41c10c602b306bfb348b5920f0efd8d00b2ca260e4b92" Nov 03 20:53:53.198931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.198902 5932 scope.go:122] "RemoveContainer" containerID="f8df41fe39f2756fd5dda64f245a9ca435e3616c8ee7c069e88e977e7fc537ea" Nov 03 20:53:53.211279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.211251 5932 scope.go:122] "RemoveContainer" containerID="ee373e99270323853147fb1fcb32dc278abc40ecd1bc3f13ac3c2767df470af3" Nov 03 20:53:53.248901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.248878 5932 scope.go:122] "RemoveContainer" containerID="3a218c2227e2cce9ca837dde6c4407ba2a9be7da1756e69232280938a15f0487" Nov 03 20:53:53.261979 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.261960 5932 scope.go:122] "RemoveContainer" containerID="578a3c9294144945ceeb56f67fcd400d1927a692916f065c3428268f0b90720b" Nov 03 20:53:53.274048 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.274014 5932 scope.go:122] "RemoveContainer" containerID="b30934b0dc85470dd47493d7af7728ec020e30503d1174ff96f2656cdaaf2982" Nov 03 20:53:53.460129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.460086 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:53.460262 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:53.460168 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:54.460667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:54.460389 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Nov 03 20:53:54.460667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:54.460450 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Nov 03 20:53:56.196905 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:56.196860 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:56.197745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:56.197470 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:56.198522 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:56.198497 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 20:53:58.821648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:58.815526 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:58.951372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:58.951325 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:59.513789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.513723 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:59.548760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.548715 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:59.817823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.817713 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:59.818828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.818804 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:53:59.895781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.895731 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:59.897983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.896365 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:53:59.898835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.896397 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": context deadline exceeded" Nov 03 20:53:59.899010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.896426 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:00.030649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.967723 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:00.030649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:53:59.968181 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:54:00.912651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:00.860858 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 20:54:00.912651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:00.861231 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 20:54:00.924399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:00.924349 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:54:00.968982 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:00.968929 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 20:54:01.053862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.031601 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:54:01.476648 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:54:01.450789 5932 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.391s" Nov 03 20:54:01.507358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.507326 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:54:01.507787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.507764 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:54:01.508343 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.508322 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:54:01.508546 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.508530 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:54:01.531284 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.508732 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:54:01.872704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.870748 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 20:54:01.872704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.871676 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:01.910651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.886064 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:01.932719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.932666 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:54:01.979793 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:01.979741 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:02.019250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.019200 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:02.105309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.105267 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:54:02.152649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.146851 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:54:02.878785 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.878735 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:02.919875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.911137 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:54:02.920088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.919944 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:54:02.920088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.919985 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:54:02.923234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.918318 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:02.944067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.944036 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:54:02.987688 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:02.986483 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:54:08.149178 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.149139 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz"] Nov 03 20:54:08.159083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.159042 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.207846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.207812 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz"] Nov 03 20:54:08.246877 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.246837 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:08.247157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.246983 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:54:08.259177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.259133 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.259355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.259263 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmzb\" (UniqueName: \"kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.259355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.259328 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.362076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.362038 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmzb\" (UniqueName: \"kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.362338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.362129 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.362491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.362465 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.365677 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.365644 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.373302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.373267 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.422713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.422637 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmzb\" (UniqueName: \"kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb\") pod \"maas-api-revoke-sub-37b51cfa-qmtlz\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.478239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.478202 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:08.867189 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.867158 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz"] Nov 03 20:54:08.867669 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:54:08.867638 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c27f29b_89ad_491c_83cf_7678d518fa95.slice/crio-305e3e9e087275ee56527e7e9ff88aa263dd99e3b00451e12de779cac42d2438 WatchSource:0}: Error finding container 305e3e9e087275ee56527e7e9ff88aa263dd99e3b00451e12de779cac42d2438: Status 404 returned error can't find the container with id 305e3e9e087275ee56527e7e9ff88aa263dd99e3b00451e12de779cac42d2438 Nov 03 20:54:08.958173 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:08.958132 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" event={"ID":"8c27f29b-89ad-491c-83cf-7678d518fa95","Type":"ContainerStarted","Data":"305e3e9e087275ee56527e7e9ff88aa263dd99e3b00451e12de779cac42d2438"} Nov 03 20:54:09.971232 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:09.971200 5932 generic.go:356] "Generic (PLEG): container finished" podID="8c27f29b-89ad-491c-83cf-7678d518fa95" containerID="03ee7826dd4183c4d3ec4d7a034ccd0a01a1ca660d36f6be693aa43d986e9d60" exitCode=0 Nov 03 20:54:09.971651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:09.971287 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" event={"ID":"8c27f29b-89ad-491c-83cf-7678d518fa95","Type":"ContainerDied","Data":"03ee7826dd4183c4d3ec4d7a034ccd0a01a1ca660d36f6be693aa43d986e9d60"} Nov 03 20:54:11.166525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.166487 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:11.207367 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.207337 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca\") pod \"8c27f29b-89ad-491c-83cf-7678d518fa95\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " Nov 03 20:54:11.207548 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.207404 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb\" (UniqueName: \"kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb\") pod \"8c27f29b-89ad-491c-83cf-7678d518fa95\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " Nov 03 20:54:11.207548 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.207438 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca\") pod \"8c27f29b-89ad-491c-83cf-7678d518fa95\" (UID: \"8c27f29b-89ad-491c-83cf-7678d518fa95\") " Nov 03 20:54:11.207741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.207716 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca" pod "8c27f29b-89ad-491c-83cf-7678d518fa95" (UID: "8c27f29b-89ad-491c-83cf-7678d518fa95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:11.208147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.208119 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:11.209656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.209611 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca" pod "8c27f29b-89ad-491c-83cf-7678d518fa95" (UID: "8c27f29b-89ad-491c-83cf-7678d518fa95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:11.209753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.209667 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb" pod "8c27f29b-89ad-491c-83cf-7678d518fa95" (UID: "8c27f29b-89ad-491c-83cf-7678d518fa95"). InnerVolumeSpecName "kube-api-access-fdmzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:11.310254 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.310173 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdmzb\" (UniqueName: \"kubernetes.io/projected/8c27f29b-89ad-491c-83cf-7678d518fa95-kube-api-access-fdmzb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:11.310254 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.310216 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c27f29b-89ad-491c-83cf-7678d518fa95-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:11.993412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.993377 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" event={"ID":"8c27f29b-89ad-491c-83cf-7678d518fa95","Type":"ContainerDied","Data":"305e3e9e087275ee56527e7e9ff88aa263dd99e3b00451e12de779cac42d2438"} Nov 03 20:54:11.993412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.993412 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz" Nov 03 20:54:11.993650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:11.993431 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="305e3e9e087275ee56527e7e9ff88aa263dd99e3b00451e12de779cac42d2438" Nov 03 20:54:14.153911 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:54:14.153869 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:54:36.866792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:36.866727 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:37.370645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:37.364905 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:44.120806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:44.120756 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:44.121331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:44.120840 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:44.156035 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:54:44.155994 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:54:45.754990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.754952 5932 generic.go:356] "Generic (PLEG): container finished" podID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerID="ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8" exitCode=0 Nov 03 20:54:45.755579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.755025 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" event={"ID":"917665c3-87ad-4ad0-b662-8358ea49e3a1","Type":"ContainerDied","Data":"ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8"} Nov 03 20:54:45.755579 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.755530 5932 scope.go:122] "RemoveContainer" containerID="ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8" Nov 03 20:54:45.757149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.757127 5932 generic.go:356] "Generic (PLEG): container finished" podID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerID="0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6" exitCode=0 Nov 03 20:54:45.757243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.757191 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" event={"ID":"21be1d12-e5f0-4161-a549-6b3a7e9ced2b","Type":"ContainerDied","Data":"0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6"} Nov 03 20:54:45.757501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.757476 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:45.757556 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.757542 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:45.757594 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.757580 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:45.757594 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.757582 5932 scope.go:122] "RemoveContainer" containerID="0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6" Nov 03 20:54:45.759304 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.759272 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:45.759421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.759354 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:45.759421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:45.759398 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:46.685413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.685374 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6"] Nov 03 20:54:46.686260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.686242 5932 cpu_manager.go:414] "containerMap: removing container" podUID="8c27f29b-89ad-491c-83cf-7678d518fa95" containerName="revoke-keys" Nov 03 20:54:46.686260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.686260 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c27f29b-89ad-491c-83cf-7678d518fa95" containerName="revoke-keys" Nov 03 20:54:46.686410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.686400 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c27f29b-89ad-491c-83cf-7678d518fa95" containerName="revoke-keys" Nov 03 20:54:46.689443 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.689426 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.718737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.718708 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j"] Nov 03 20:54:46.742656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.741950 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805378 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805441 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805504 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805570 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805605 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805655 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805694 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805748 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805775 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805887 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805917 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.805978 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.806039 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgnqg\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.806080 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.806127 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5wp\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.806175 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.806234 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.807358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.806267 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.816671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.816597 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6"] Nov 03 20:54:46.839661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.821155 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" event={"ID":"21be1d12-e5f0-4161-a549-6b3a7e9ced2b","Type":"ContainerStarted","Data":"faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe"} Nov 03 20:54:46.861664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.850497 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j"] Nov 03 20:54:46.897770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.896809 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" event={"ID":"917665c3-87ad-4ad0-b662-8358ea49e3a1","Type":"ContainerStarted","Data":"14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1"} Nov 03 20:54:46.907140 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907110 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:54:46.907385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907369 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:46.907511 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907494 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:46.907815 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907793 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5wp\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.907901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907843 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.907901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907880 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907904 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907958 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908014 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.907979 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908021 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908066 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908086 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908104 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908128 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908167 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908193 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908278 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908297 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908337 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908338 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908430 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgnqg\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908489 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908482 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908877 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908520 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.908877 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.908830 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.909119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.909087 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.909214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.909123 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.909355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.909333 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.909735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.909710 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.909836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.909732 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.909999 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.909976 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.910124 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.910104 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.912021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.911995 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.912581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.912554 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.913719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.913676 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:46.915205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.915186 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.920467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.920448 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:54:46.953704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.951772 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:46.953704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.951838 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:46.989333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.988843 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:46.997646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:46.996796 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5wp\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-njdm6\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:47.005897 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.005797 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:47.006349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.006323 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:47.006822 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.006801 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgnqg\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-66t8j\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:47.111333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.111295 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:47.641830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.631739 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6"] Nov 03 20:54:47.641830 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.641647 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j"] Nov 03 20:54:47.691562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.652235 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:47.691562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.652339 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:47.691562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.652389 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:47.715869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.715826 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:47.716023 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.715957 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:47.716092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.716011 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:54:47.909153 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.909052 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:47.909153 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.909107 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:47.948648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.928180 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" event={"ID":"81f419fb-6c7a-4922-9f0c-ceb825172fd2","Type":"ContainerStarted","Data":"f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673"} Nov 03 20:54:47.948648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.928245 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" event={"ID":"81f419fb-6c7a-4922-9f0c-ceb825172fd2","Type":"ContainerStarted","Data":"5f00d365a569e75573e2226f5b05c64cd81ee95c36078a50d5f28a7767c571fd"} Nov 03 20:54:47.973646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.956207 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:47.973646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.956274 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:48.003680 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.999018 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" event={"ID":"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86","Type":"ContainerStarted","Data":"9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06"} Nov 03 20:54:48.003680 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:47.999075 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" event={"ID":"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86","Type":"ContainerStarted","Data":"1a364074fafbc69031e31ce0a27681b43b78fbfd3b198d83929d3aa385a20bea"} Nov 03 20:54:48.007856 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.007829 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:48.008010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.007939 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:48.008010 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.007984 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:48.062730 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.053989 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podStartSLOduration=2.053965821 podStartE2EDuration="2.053965821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:48.045527014 +0000 UTC m=+1496.110768488" watchObservedRunningTime="2026-11-03 20:54:48.053965821 +0000 UTC m=+1496.119207374" Nov 03 20:54:48.112000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.111964 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:48.113717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.113686 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:48.113813 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.113735 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:48.183738 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.183453 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podStartSLOduration=2.183433472 podStartE2EDuration="2.183433472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:48.177671599 +0000 UTC m=+1496.242913075" watchObservedRunningTime="2026-11-03 20:54:48.183433472 +0000 UTC m=+1496.248674948" Nov 03 20:54:48.917646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.917092 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:48.917646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.917155 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:48.938648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.929052 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:48.938648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:48.929114 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:49.020660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.020380 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:49.020660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.020446 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:49.124646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.112125 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:49.124646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.112188 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:49.859650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.858890 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:49.879648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.860092 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:49.886658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.886440 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:49.923666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.912031 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:49.923666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.912095 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:49.931647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.931467 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:49.931647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:49.931528 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:50.012376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.010053 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:50.012376 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.010118 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:50.024746 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.014733 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:50.035658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.035410 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 20:54:50.112560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.111355 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:54:50.112560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.112396 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:50.112560 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.112436 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:50.164767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.164720 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:50.349647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.347729 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:54:50.441355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.441316 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:54:50.908647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.908498 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:50.908647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.908564 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:50.933649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.931006 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:50.933649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:50.931070 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:51.017710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.015812 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:51.017710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.015880 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:51.120643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.120125 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:51.120643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.120202 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:51.372702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.371892 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:51.391651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.373605 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:54:51.899646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.878660 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:51.899646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.879004 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 20:54:51.900079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.900024 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:51.992042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.990735 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:51.992042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:51.990799 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:52.030648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.029709 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:52.109649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.054759 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:52.109649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.101239 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:52.109649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.101304 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:52.109649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.102127 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:52.109649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.102167 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:52.111947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.111681 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 20:54:52.113417 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.113389 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:52.113588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.113562 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:52.196705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.154718 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:54:52.665648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.664855 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 20:54:52.920702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.904996 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:52.935661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.921276 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 20:54:52.936225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.925444 5932 patch_prober.go:29] interesting pod/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 20:54:52.936447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.936419 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 20:54:52.960861 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.927144 5932 patch_prober.go:29] interesting pod/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 20:54:53.005674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.961081 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 20:54:53.063667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:52.973289 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:54:53.063667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.060826 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:54:53.063667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.034507 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:53.063667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.062011 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:53.063667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.059164 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:53.063667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.063222 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:54:53.064131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.036633 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:53.064131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.063987 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:54:53.102715 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.102671 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8"] Nov 03 20:54:53.113016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.112539 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:53.113016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.112600 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:53.113016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.112696 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:53.172752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.158065 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:53.325642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.258098 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:54:53.325642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.300974 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:54:53.325642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.258311 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:53.325642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.324861 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8"] Nov 03 20:54:53.683968 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.683927 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:53.719645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.715594 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 20:54:53.724324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.724299 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 20:54:53.724528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.724517 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 20:54:53.753897 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.749113 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 20:54:53.770252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.770226 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:54:53.771847 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.771825 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:54:53.772172 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.772143 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 20:54:53.797674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.791781 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:53.797674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.791881 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7d6\" (UniqueName: \"kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:53.797674 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.791996 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:53.801609 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.801577 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 20:54:53.941056 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.941022 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 20:54:53.992519 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:53.992482 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:54:54.019380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.019166 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:54:54.059569 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.059531 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:54:54.302696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.299134 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:54.302696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.299200 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:54.302696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.299568 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:54:54.302696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.299832 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:54:54.302696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.299921 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:54.302696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.299958 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:54.718652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.717666 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 20:54:54.754120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.744846 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:54.754120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.745464 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:54.754120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.745596 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7d6\" (UniqueName: \"kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:54.754120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.751583 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:54.903246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.903208 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:54.969954 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:54.969849 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7d6\" (UniqueName: \"kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6\") pod \"maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:55.030864 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.016019 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:55.057528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.036995 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:55.215776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.199672 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:55.215776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.199734 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:55.215776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.205984 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:55.888193 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.872555 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:54:55.888193 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:55.875252 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:54:56.009013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.008968 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:56.009215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.009035 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:56.112110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.111899 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:56.112110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.111960 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:56.178658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.172193 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8"] Nov 03 20:54:56.906075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.906036 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" event={"ID":"ca986db2-b1de-47ed-a4b5-5ab795d1b764","Type":"ContainerStarted","Data":"4c8e69f91117c583a1a3929f93934b3ca99877c8969c7a8441f60d6da90143e9"} Nov 03 20:54:56.906289 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:56.906137 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" event={"ID":"ca986db2-b1de-47ed-a4b5-5ab795d1b764","Type":"ContainerStarted","Data":"2b094c9f7e5fde40d237e28a09533123337d15053c3721a2944f2e7285a08c6e"} Nov 03 20:54:57.008775 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.008727 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 20:54:57.009100 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.009063 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 20:54:57.025646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.022542 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" podStartSLOduration=6.022523196 podStartE2EDuration="6.022523196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:57.016086857 +0000 UTC m=+1505.081328332" watchObservedRunningTime="2026-11-03 20:54:57.022523196 +0000 UTC m=+1505.087764670" Nov 03 20:54:57.112990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.112946 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 20:54:57.113400 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.113023 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 20:54:57.919931 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.919891 5932 generic.go:356] "Generic (PLEG): container finished" podID="ca986db2-b1de-47ed-a4b5-5ab795d1b764" containerID="4c8e69f91117c583a1a3929f93934b3ca99877c8969c7a8441f60d6da90143e9" exitCode=0 Nov 03 20:54:57.920212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:57.920192 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" event={"ID":"ca986db2-b1de-47ed-a4b5-5ab795d1b764","Type":"ContainerDied","Data":"4c8e69f91117c583a1a3929f93934b3ca99877c8969c7a8441f60d6da90143e9"} Nov 03 20:54:58.101686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:58.100352 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z"] Nov 03 20:54:58.110530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:58.110354 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-236db804-ccl5z"] Nov 03 20:54:59.056461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.055753 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:59.056461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.055841 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:59.174177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.153872 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:54:59.174177 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.153978 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:54:59.285586 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.285221 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:59.286229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.286205 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:59.290998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.290975 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 20:54:59.309543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.309504 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:59.310694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.310673 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:59.312073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.312049 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 20:54:59.317922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.317897 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:59.381035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.380967 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca\") pod \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " Nov 03 20:54:59.381255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.381221 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6\" (UniqueName: \"kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6\") pod \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " Nov 03 20:54:59.381918 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.381889 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca\") pod \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\" (UID: \"ca986db2-b1de-47ed-a4b5-5ab795d1b764\") " Nov 03 20:54:59.382343 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.382317 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca" pod "ca986db2-b1de-47ed-a4b5-5ab795d1b764" (UID: "ca986db2-b1de-47ed-a4b5-5ab795d1b764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:54:59.383246 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.383221 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:59.383596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.383564 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca" pod "ca986db2-b1de-47ed-a4b5-5ab795d1b764" (UID: "ca986db2-b1de-47ed-a4b5-5ab795d1b764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:54:59.383769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.383747 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6" pod "ca986db2-b1de-47ed-a4b5-5ab795d1b764" (UID: "ca986db2-b1de-47ed-a4b5-5ab795d1b764"). InnerVolumeSpecName "kube-api-access-8f7d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:54:59.504654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.493223 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca986db2-b1de-47ed-a4b5-5ab795d1b764-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:59.504654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.493265 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f7d6\" (UniqueName: \"kubernetes.io/projected/ca986db2-b1de-47ed-a4b5-5ab795d1b764-kube-api-access-8f7d6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:54:59.745510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.745474 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh"] Nov 03 20:54:59.746917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.746890 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ca986db2-b1de-47ed-a4b5-5ab795d1b764" containerName="revoke-keys" Nov 03 20:54:59.746917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.746913 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca986db2-b1de-47ed-a4b5-5ab795d1b764" containerName="revoke-keys" Nov 03 20:54:59.747085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.747051 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca986db2-b1de-47ed-a4b5-5ab795d1b764" containerName="revoke-keys" Nov 03 20:54:59.751311 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.751284 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.798428 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.798393 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh"] Nov 03 20:54:59.821616 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.821580 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86pqk\" (UniqueName: \"kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.821808 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.821771 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.821881 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.821820 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.923279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.923241 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86pqk\" (UniqueName: \"kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.923476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.923458 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.923564 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.923539 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.924458 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.924429 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.926029 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.926002 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.980774 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.980734 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86pqk\" (UniqueName: \"kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk\") pod \"maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:54:59.996375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.996301 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" Nov 03 20:54:59.996375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.996336 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8" event={"ID":"ca986db2-b1de-47ed-a4b5-5ab795d1b764","Type":"ContainerDied","Data":"2b094c9f7e5fde40d237e28a09533123337d15053c3721a2944f2e7285a08c6e"} Nov 03 20:54:59.996576 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:54:59.996386 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b094c9f7e5fde40d237e28a09533123337d15053c3721a2944f2e7285a08c6e" Nov 03 20:55:00.059905 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:00.059870 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67f0a98-41c3-4222-af2d-99d5a986f4df" path="/var/lib/kubelet/pods/e67f0a98-41c3-4222-af2d-99d5a986f4df/volumes" Nov 03 20:55:00.073058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:00.073014 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:55:00.506292 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:00.506265 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh"] Nov 03 20:55:00.508563 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:55:00.508533 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24c4f64c_c7d1_4e92_9de2_43705a9b367c.slice/crio-904ebbd73398186b9c43217d045bfd362a2ed2194df3f1bc764fdbc7931c2f70 WatchSource:0}: Error finding container 904ebbd73398186b9c43217d045bfd362a2ed2194df3f1bc764fdbc7931c2f70: Status 404 returned error can't find the container with id 904ebbd73398186b9c43217d045bfd362a2ed2194df3f1bc764fdbc7931c2f70 Nov 03 20:55:01.010885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:01.010841 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" event={"ID":"24c4f64c-c7d1-4e92-9de2-43705a9b367c","Type":"ContainerStarted","Data":"26969f472eac18a47c8dedf9171d5ab0350ca883c27c42a872ee74122e5dc36b"} Nov 03 20:55:01.011057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:01.010950 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" event={"ID":"24c4f64c-c7d1-4e92-9de2-43705a9b367c","Type":"ContainerStarted","Data":"904ebbd73398186b9c43217d045bfd362a2ed2194df3f1bc764fdbc7931c2f70"} Nov 03 20:55:01.123068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:01.123013 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" podStartSLOduration=2.122997516 podStartE2EDuration="2.122997516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:01.119775238 +0000 UTC m=+1509.185016711" watchObservedRunningTime="2026-11-03 20:55:01.122997516 +0000 UTC m=+1509.188239013" Nov 03 20:55:02.021800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:02.021716 5932 generic.go:356] "Generic (PLEG): container finished" podID="24c4f64c-c7d1-4e92-9de2-43705a9b367c" containerID="26969f472eac18a47c8dedf9171d5ab0350ca883c27c42a872ee74122e5dc36b" exitCode=0 Nov 03 20:55:02.021939 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:02.021814 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" event={"ID":"24c4f64c-c7d1-4e92-9de2-43705a9b367c","Type":"ContainerDied","Data":"26969f472eac18a47c8dedf9171d5ab0350ca883c27c42a872ee74122e5dc36b"} Nov 03 20:55:03.213778 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.213750 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:55:03.280916 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.280881 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca\") pod \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " Nov 03 20:55:03.281129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.281110 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk\" (UniqueName: \"kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk\") pod \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " Nov 03 20:55:03.281179 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.281153 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca\") pod \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\" (UID: \"24c4f64c-c7d1-4e92-9de2-43705a9b367c\") " Nov 03 20:55:03.281310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.281288 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca" pod "24c4f64c-c7d1-4e92-9de2-43705a9b367c" (UID: "24c4f64c-c7d1-4e92-9de2-43705a9b367c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:03.281922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.281856 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:03.283827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.283606 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca" pod "24c4f64c-c7d1-4e92-9de2-43705a9b367c" (UID: "24c4f64c-c7d1-4e92-9de2-43705a9b367c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:03.283827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.283769 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk" pod "24c4f64c-c7d1-4e92-9de2-43705a9b367c" (UID: "24c4f64c-c7d1-4e92-9de2-43705a9b367c"). InnerVolumeSpecName "kube-api-access-86pqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:03.385682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.384269 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86pqk\" (UniqueName: \"kubernetes.io/projected/24c4f64c-c7d1-4e92-9de2-43705a9b367c-kube-api-access-86pqk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:03.385682 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.384311 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24c4f64c-c7d1-4e92-9de2-43705a9b367c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:03.661698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:03.661658 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng"] Nov 03 20:55:04.127827 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.127339 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" Nov 03 20:55:04.129967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.129929 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh" event={"ID":"24c4f64c-c7d1-4e92-9de2-43705a9b367c","Type":"ContainerDied","Data":"904ebbd73398186b9c43217d045bfd362a2ed2194df3f1bc764fdbc7931c2f70"} Nov 03 20:55:04.130097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.130006 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="904ebbd73398186b9c43217d045bfd362a2ed2194df3f1bc764fdbc7931c2f70" Nov 03 20:55:04.145949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.144541 5932 generic.go:356] "Generic (PLEG): container finished" podID="2145eea5-0bc2-4812-9538-b6325b622e38" containerID="b4daa8080a00865c328a1ec4351f965060499399fd2601731866b5347480e86a" exitCode=0 Nov 03 20:55:04.145949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.144878 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" event={"ID":"2145eea5-0bc2-4812-9538-b6325b622e38","Type":"ContainerDied","Data":"b4daa8080a00865c328a1ec4351f965060499399fd2601731866b5347480e86a"} Nov 03 20:55:04.145949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.145326 5932 scope.go:122] "RemoveContainer" containerID="b4daa8080a00865c328a1ec4351f965060499399fd2601731866b5347480e86a" Nov 03 20:55:04.148091 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.148053 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:04.148218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.148148 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:04.148218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:04.148199 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:05.205654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:05.205087 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" event={"ID":"2145eea5-0bc2-4812-9538-b6325b622e38","Type":"ContainerStarted","Data":"70825b91580b2f3822fafe3e164e40e4e9687573059ec00cb6d129717e3bc683"} Nov 03 20:55:05.994359 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:05.994325 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:55:05.994595 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:05.994569 5932 patch_prober.go:29] interesting pod/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:55:05.994722 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:05.994670 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:55:06.998734 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:06.998693 5932 patch_prober.go:29] interesting pod/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:55:06.999354 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:06.999318 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:55:07.994476 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:07.994444 5932 patch_prober.go:29] interesting pod/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:55:07.994808 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:07.994497 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:55:08.662556 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:08.662500 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" podUID="ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" containerName="payload-pre-processing" containerID="cri-o://68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57" gracePeriod=25 Nov 03 20:55:08.994285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:08.994250 5932 patch_prober.go:29] interesting pod/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:55:08.994413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:08.994313 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:55:09.002355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.002325 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:55:09.089679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.089607 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume\") pod \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " Nov 03 20:55:09.089873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.089780 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb\" (UniqueName: \"kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb\") pod \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\" (UID: \"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e\") " Nov 03 20:55:09.090062 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.090037 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume" pod "ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" (UID: "ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:09.090500 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.090475 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:09.092260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.092232 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb" pod "ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" (UID: "ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e"). InnerVolumeSpecName "kube-api-access-mgpjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:09.192716 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.192679 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgpjb\" (UniqueName: \"kubernetes.io/projected/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e-kube-api-access-mgpjb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:09.266260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.266167 5932 generic.go:356] "Generic (PLEG): container finished" podID="ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" containerID="68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57" exitCode=0 Nov 03 20:55:09.266260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.266218 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" Nov 03 20:55:09.266260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.266246 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" event={"ID":"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e","Type":"ContainerDied","Data":"68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57"} Nov 03 20:55:09.266510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.266315 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng" event={"ID":"ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e","Type":"ContainerDied","Data":"89a88e3645933504622c7a92965260d9cebcee219bd692ac56b6ee9166a6466a"} Nov 03 20:55:09.266510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.266341 5932 scope.go:122] "RemoveContainer" containerID="68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57" Nov 03 20:55:09.282414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.282388 5932 scope.go:122] "RemoveContainer" containerID="68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57" Nov 03 20:55:09.283029 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:55:09.283004 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57\": container with ID starting with 68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57 not found: ID does not exist" containerID="68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57" Nov 03 20:55:09.283124 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.283037 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57"} err="failed to get container status \"68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57\": rpc error: code = NotFound desc = could not find container \"68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57\": container with ID starting with 68a22c06624544840c155595c5821463dd879751bbf771a4efb8cead43517f57 not found: ID does not exist" Nov 03 20:55:09.447710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.447335 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng"] Nov 03 20:55:09.453980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.453946 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-b6eaffa8-6f6cb9b679-kn7ng"] Nov 03 20:55:09.994535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.994492 5932 patch_prober.go:29] interesting pod/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:55:09.994985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:09.994570 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:55:10.056058 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.056022 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" path="/var/lib/kubelet/pods/ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e/volumes" Nov 03 20:55:10.437758 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.436700 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv"] Nov 03 20:55:10.437971 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.437879 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" containerName="payload-pre-processing" Nov 03 20:55:10.437971 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.437897 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" containerName="payload-pre-processing" Nov 03 20:55:10.437971 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.437920 5932 cpu_manager.go:414] "containerMap: removing container" podUID="24c4f64c-c7d1-4e92-9de2-43705a9b367c" containerName="revoke-keys" Nov 03 20:55:10.437971 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.437928 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24c4f64c-c7d1-4e92-9de2-43705a9b367c" containerName="revoke-keys" Nov 03 20:55:10.438279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.438166 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5dc2c2-a7ca-46d4-b33c-c368cdc6375e" containerName="payload-pre-processing" Nov 03 20:55:10.438279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.438179 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="24c4f64c-c7d1-4e92-9de2-43705a9b367c" containerName="revoke-keys" Nov 03 20:55:10.552823 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.552778 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv"] Nov 03 20:55:10.553020 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.552883 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.602047 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.602011 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:55:10.602213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.602106 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:55:10.646944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.646905 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.647147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.647118 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.647213 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.647146 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnlg\" (UniqueName: \"kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.751687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.751373 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.751687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.751564 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.751687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.751598 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnlg\" (UniqueName: \"kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.754648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.752433 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.754648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.754589 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.809352 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.809313 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnlg\" (UniqueName: \"kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg\") pod \"maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.868332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.868294 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:10.994670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.994165 5932 patch_prober.go:29] interesting pod/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 20:55:10.994670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:10.994221 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 20:55:11.280035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:11.280000 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv"] Nov 03 20:55:11.998779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:11.998702 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:55:11.999206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:11.999184 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:55:12.000024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:12.000008 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:55:12.304131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:12.304036 5932 generic.go:356] "Generic (PLEG): container finished" podID="02b969eb-b724-4f53-8685-507c1c1fc9c9" containerID="66649da2eff3f903a465fd436067e61e8066c9f1ad71eb040ed6f4111a496b29" exitCode=0 Nov 03 20:55:12.305523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:12.305488 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" event={"ID":"02b969eb-b724-4f53-8685-507c1c1fc9c9","Type":"ContainerDied","Data":"66649da2eff3f903a465fd436067e61e8066c9f1ad71eb040ed6f4111a496b29"} Nov 03 20:55:12.305679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:12.305545 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" event={"ID":"02b969eb-b724-4f53-8685-507c1c1fc9c9","Type":"ContainerStarted","Data":"0f5e43bb04c05194c8914beecb37f294ad9f859619ba8ac81f1d09677d57d56a"} Nov 03 20:55:13.597160 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.597133 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:13.658296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.658215 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca\") pod \"02b969eb-b724-4f53-8685-507c1c1fc9c9\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " Nov 03 20:55:13.658296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.658262 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg\" (UniqueName: \"kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg\") pod \"02b969eb-b724-4f53-8685-507c1c1fc9c9\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " Nov 03 20:55:13.658482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.658301 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca\") pod \"02b969eb-b724-4f53-8685-507c1c1fc9c9\" (UID: \"02b969eb-b724-4f53-8685-507c1c1fc9c9\") " Nov 03 20:55:13.658807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.658773 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca" pod "02b969eb-b724-4f53-8685-507c1c1fc9c9" (UID: "02b969eb-b724-4f53-8685-507c1c1fc9c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:13.658947 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.658932 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:13.660478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.660442 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg" pod "02b969eb-b724-4f53-8685-507c1c1fc9c9" (UID: "02b969eb-b724-4f53-8685-507c1c1fc9c9"). InnerVolumeSpecName "kube-api-access-cwnlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:13.660594 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.660479 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca" pod "02b969eb-b724-4f53-8685-507c1c1fc9c9" (UID: "02b969eb-b724-4f53-8685-507c1c1fc9c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:13.761486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.761448 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02b969eb-b724-4f53-8685-507c1c1fc9c9-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:13.761486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:13.761486 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwnlg\" (UniqueName: \"kubernetes.io/projected/02b969eb-b724-4f53-8685-507c1c1fc9c9-kube-api-access-cwnlg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:14.327977 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:55:14.326082 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:55:14.504769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:14.504733 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" event={"ID":"02b969eb-b724-4f53-8685-507c1c1fc9c9","Type":"ContainerDied","Data":"0f5e43bb04c05194c8914beecb37f294ad9f859619ba8ac81f1d09677d57d56a"} Nov 03 20:55:14.505081 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:14.505063 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f5e43bb04c05194c8914beecb37f294ad9f859619ba8ac81f1d09677d57d56a" Nov 03 20:55:14.505547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:14.505233 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv" Nov 03 20:55:30.984409 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:30.984375 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk"] Nov 03 20:55:30.985212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:30.985196 5932 cpu_manager.go:414] "containerMap: removing container" podUID="02b969eb-b724-4f53-8685-507c1c1fc9c9" containerName="revoke-keys" Nov 03 20:55:30.985279 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:30.985214 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02b969eb-b724-4f53-8685-507c1c1fc9c9" containerName="revoke-keys" Nov 03 20:55:30.985367 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:30.985356 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="02b969eb-b724-4f53-8685-507c1c1fc9c9" containerName="revoke-keys" Nov 03 20:55:30.990933 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:30.990915 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.018918 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.018891 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsgl\" (UniqueName: \"kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.019136 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.019115 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.019196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.019145 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.045725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.045692 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk"] Nov 03 20:55:31.083083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.083046 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:55:31.083331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.083171 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:55:31.120255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.120218 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsgl\" (UniqueName: \"kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.120410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.120380 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.120410 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.120402 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.121106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.121081 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.122930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.122907 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.178203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.178173 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsgl\" (UniqueName: \"kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl\") pod \"maas-api-revoke-sub-0f518bcb-qt5tk\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.305844 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.305756 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:31.702040 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.702012 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk"] Nov 03 20:55:31.704863 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:55:31.704827 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d2586e_63e1_4eb9_9df4_5a2ea1de15f2.slice/crio-588d770acf9cd7d5829f10c4e850f980074b041e1165aa486dd0fbb4aba3e829 WatchSource:0}: Error finding container 588d770acf9cd7d5829f10c4e850f980074b041e1165aa486dd0fbb4aba3e829: Status 404 returned error can't find the container with id 588d770acf9cd7d5829f10c4e850f980074b041e1165aa486dd0fbb4aba3e829 Nov 03 20:55:31.859727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:31.859693 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" event={"ID":"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2","Type":"ContainerStarted","Data":"588d770acf9cd7d5829f10c4e850f980074b041e1165aa486dd0fbb4aba3e829"} Nov 03 20:55:32.871261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:32.871226 5932 generic.go:356] "Generic (PLEG): container finished" podID="19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" containerID="32c2b2d696168bcf020c5794ba0aec9eab9e9643b329780fb6144b98fae06247" exitCode=0 Nov 03 20:55:32.871713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:32.871309 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" event={"ID":"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2","Type":"ContainerDied","Data":"32c2b2d696168bcf020c5794ba0aec9eab9e9643b329780fb6144b98fae06247"} Nov 03 20:55:34.074457 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.074431 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:34.196843 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.196752 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca\") pod \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " Nov 03 20:55:34.197050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.196930 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl\" (UniqueName: \"kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl\") pod \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " Nov 03 20:55:34.197050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.196962 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca\") pod \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\" (UID: \"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2\") " Nov 03 20:55:34.197537 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.197505 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca" pod "19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" (UID: "19d2586e-63e1-4eb9-9df4-5a2ea1de15f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:34.197819 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.197789 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.199836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.199810 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca" pod "19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" (UID: "19d2586e-63e1-4eb9-9df4-5a2ea1de15f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:55:34.208902 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.208862 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl" pod "19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" (UID: "19d2586e-63e1-4eb9-9df4-5a2ea1de15f2"). InnerVolumeSpecName "kube-api-access-vwsgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:34.299801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.299764 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.299801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.299798 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsgl\" (UniqueName: \"kubernetes.io/projected/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2-kube-api-access-vwsgl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:34.892057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.892029 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" Nov 03 20:55:34.892291 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.892060 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk" event={"ID":"19d2586e-63e1-4eb9-9df4-5a2ea1de15f2","Type":"ContainerDied","Data":"588d770acf9cd7d5829f10c4e850f980074b041e1165aa486dd0fbb4aba3e829"} Nov 03 20:55:34.892291 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:34.892119 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="588d770acf9cd7d5829f10c4e850f980074b041e1165aa486dd0fbb4aba3e829" Nov 03 20:55:39.051894 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:39.051856 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85"] Nov 03 20:55:39.097431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:39.097395 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c955f2ea-87f85"] Nov 03 20:55:40.053642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:40.053596 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a819c87e-860b-4538-8918-d8b6515cbba8" path="/var/lib/kubelet/pods/a819c87e-860b-4538-8918-d8b6515cbba8/volumes" Nov 03 20:55:44.152807 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:55:44.152765 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:55:45.662647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:45.651253 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr"] Nov 03 20:55:45.662647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:45.653127 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" containerID="cri-o://faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe" gracePeriod=30 Nov 03 20:55:45.704252 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:45.704213 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8"] Nov 03 20:55:45.710872 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:45.710839 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-b6eaffa8-2zgp8"] Nov 03 20:55:46.152012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.151974 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca986db2-b1de-47ed-a4b5-5ab795d1b764" path="/var/lib/kubelet/pods/ca986db2-b1de-47ed-a4b5-5ab795d1b764/volumes" Nov 03 20:55:46.717833 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.717792 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh"] Nov 03 20:55:46.718977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.718951 5932 cpu_manager.go:414] "containerMap: removing container" podUID="19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" containerName="revoke-keys" Nov 03 20:55:46.718977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.718974 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" containerName="revoke-keys" Nov 03 20:55:46.719167 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.719157 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" containerName="revoke-keys" Nov 03 20:55:46.723384 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.723362 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.775462 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.775415 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-df4313ad-openshift-default-dockercfg-ljk7d\"" Nov 03 20:55:46.828016 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.827983 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh"] Nov 03 20:55:46.851239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851199 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851244 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851301 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851365 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851399 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851487 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851422 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851562 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851677 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdzn\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.851763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.851721 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.974808 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.974912 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdzn\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.974969 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975116 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975142 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975198 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975258 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975297 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975324 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975938 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.975964 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.976193 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:46.978780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:46.977038 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.054426 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.034450 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.170887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.083446 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.170887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.122109 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdzn\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.170887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.151206 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.170887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.151276 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh"] Nov 03 20:55:47.224363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.224297 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15564eb1-j4zqh"] Nov 03 20:55:47.231072 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.230990 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert\") pod \"e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.349074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.347950 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:47.516026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.515948 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2"] Nov 03 20:55:47.516552 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:47.516191 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" containerID="cri-o://70825b91580b2f3822fafe3e164e40e4e9687573059ec00cb6d129717e3bc683" gracePeriod=30 Nov 03 20:55:48.054212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:48.054172 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24c4f64c-c7d1-4e92-9de2-43705a9b367c" path="/var/lib/kubelet/pods/24c4f64c-c7d1-4e92-9de2-43705a9b367c/volumes" Nov 03 20:55:48.126082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:48.126051 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh"] Nov 03 20:55:48.129143 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:55:48.129117 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556c9d90_35c0_4dc1_a40c_8a0bf31b479b.slice/crio-59d8c57a76bd14571f5579f1b2681520719f80434a3e7ae21354d1633b06609b WatchSource:0}: Error finding container 59d8c57a76bd14571f5579f1b2681520719f80434a3e7ae21354d1633b06609b: Status 404 returned error can't find the container with id 59d8c57a76bd14571f5579f1b2681520719f80434a3e7ae21354d1633b06609b Nov 03 20:55:48.132186 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:48.132153 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:48.132271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:48.132217 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:48.132271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:48.132250 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:48.214218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:48.214189 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" event={"ID":"556c9d90-35c0-4dc1-a40c-8a0bf31b479b","Type":"ContainerStarted","Data":"59d8c57a76bd14571f5579f1b2681520719f80434a3e7ae21354d1633b06609b"} Nov 03 20:55:49.080643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.076269 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh"] Nov 03 20:55:49.091896 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.091866 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.153299 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.153265 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh"] Nov 03 20:55:49.204920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.204866 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-299a741b-openshift-default-dockercfg-7lg6b\"" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283405 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283475 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283664 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283729 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86dnz\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283764 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283855 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.283944 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.284014 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.287657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.284086 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.577463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.577396 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86dnz\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.577707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.577513 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.577707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.577655 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.577834 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.577778 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.577903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.577886 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.578002 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.577981 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.578216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.578177 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.578278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.578252 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.578613 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.578589 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.582881 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.582844 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.584725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.584695 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.585167 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.585144 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.600032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.587494 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.604247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.593323 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.621980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.620594 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.629261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.627170 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" event={"ID":"556c9d90-35c0-4dc1-a40c-8a0bf31b479b","Type":"ContainerStarted","Data":"428e073bdb16da9a03daaec67437cf83307585811c5f94ff317be973f180c62d"} Nov 03 20:55:49.638499 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.638444 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.653082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.653025 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.654662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.654583 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86dnz\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz\") pod \"e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.727903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.727868 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:49.782816 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:49.782762 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podStartSLOduration=3.782742997 podStartE2EDuration="3.782742997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:49.781115347 +0000 UTC m=+1557.846356822" watchObservedRunningTime="2026-11-03 20:55:49.782742997 +0000 UTC m=+1557.847984475" Nov 03 20:55:50.232639 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.232596 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh"] Nov 03 20:55:50.266948 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:55:50.266911 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94be4b6a_4a95_40e7_9ead_b4f02fd232c5.slice/crio-67fccc66dd4d4698e585f345d75dceee1d571d8aab451f162dc734fe048b88e0 WatchSource:0}: Error finding container 67fccc66dd4d4698e585f345d75dceee1d571d8aab451f162dc734fe048b88e0: Status 404 returned error can't find the container with id 67fccc66dd4d4698e585f345d75dceee1d571d8aab451f162dc734fe048b88e0 Nov 03 20:55:50.269741 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.269708 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:50.269828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.269783 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:50.269828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.269814 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:55:50.348874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.348843 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:50.355379 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.355349 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:50.640779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.640737 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" event={"ID":"94be4b6a-4a95-40e7-9ead-b4f02fd232c5","Type":"ContainerStarted","Data":"9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb"} Nov 03 20:55:50.640989 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.640801 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" event={"ID":"94be4b6a-4a95-40e7-9ead-b4f02fd232c5","Type":"ContainerStarted","Data":"67fccc66dd4d4698e585f345d75dceee1d571d8aab451f162dc734fe048b88e0"} Nov 03 20:55:50.641161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.641143 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:50.644141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.644114 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:55:50.729699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.729660 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:50.755267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:50.755217 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podStartSLOduration=2.755203974 podStartE2EDuration="2.755203974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:50.753194456 +0000 UTC m=+1558.818435932" watchObservedRunningTime="2026-11-03 20:55:50.755203974 +0000 UTC m=+1558.820445503" Nov 03 20:55:51.132549 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.132521 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:55:51.275909 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.275874 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276007 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276064 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276126 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276156 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276302 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276333 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276406 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276822 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276492 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.276822 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276476 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket\") pod \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\" (UID: \"21be1d12-e5f0-4161-a549-6b3a7e9ced2b\") " Nov 03 20:55:51.276822 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.276732 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.277057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277034 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:51.277268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277236 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.277435 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277418 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.277702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277681 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.277781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277708 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.277781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277722 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.277781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277736 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.277781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.277750 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.280094 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.280066 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:51.280302 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.280281 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "kube-api-access-bfjfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:51.280765 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.280741 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:51.280852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.280806 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy" pod "21be1d12-e5f0-4161-a549-6b3a7e9ced2b" (UID: "21be1d12-e5f0-4161-a549-6b3a7e9ced2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:51.379488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.379451 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.379488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.379484 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.379488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.379494 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.379744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.379503 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfjfv\" (UniqueName: \"kubernetes.io/projected/21be1d12-e5f0-4161-a549-6b3a7e9ced2b-kube-api-access-bfjfv\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:51.734229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.734086 5932 generic.go:356] "Generic (PLEG): container finished" podID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerID="faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe" exitCode=0 Nov 03 20:55:51.748326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.734660 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" event={"ID":"21be1d12-e5f0-4161-a549-6b3a7e9ced2b","Type":"ContainerDied","Data":"faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe"} Nov 03 20:55:51.748326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.734731 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" event={"ID":"21be1d12-e5f0-4161-a549-6b3a7e9ced2b","Type":"ContainerDied","Data":"5fa9e07a2bf7e9f82c3dc2e4aaa36796d4e15f885249836b6c2f01cf0205ff36"} Nov 03 20:55:51.748326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.734763 5932 scope.go:122] "RemoveContainer" containerID="faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe" Nov 03 20:55:51.748326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.734812 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr" Nov 03 20:55:51.748326 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.746795 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:51.835129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.830813 5932 scope.go:122] "RemoveContainer" containerID="0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6" Nov 03 20:55:51.865917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.865864 5932 scope.go:122] "RemoveContainer" containerID="faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe" Nov 03 20:55:51.866428 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:55:51.866393 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe\": container with ID starting with faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe not found: ID does not exist" containerID="faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe" Nov 03 20:55:51.866583 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.866556 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe"} err="failed to get container status \"faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe\": rpc error: code = NotFound desc = could not find container \"faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe\": container with ID starting with faa486662409f9fbd3f387b8a796a63a119ce034e60fa9078c5e6052fe8826fe not found: ID does not exist" Nov 03 20:55:51.866690 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.866681 5932 scope.go:122] "RemoveContainer" containerID="0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6" Nov 03 20:55:51.868725 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:55:51.867045 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6\": container with ID starting with 0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6 not found: ID does not exist" containerID="0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6" Nov 03 20:55:51.868725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:51.867076 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6"} err="failed to get container status \"0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6\": rpc error: code = NotFound desc = could not find container \"0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6\": container with ID starting with 0c92d69eb5c29f672c058ccb7f6f3f61251cc8fdc321d130e95ee33da6fd21a6 not found: ID does not exist" Nov 03 20:55:52.017955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.017867 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr"] Nov 03 20:55:52.029518 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.029484 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-b6eaffa8-gw-openshift-default-7f5976d677-dtkkr"] Nov 03 20:55:52.058239 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.058202 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" path="/var/lib/kubelet/pods/21be1d12-e5f0-4161-a549-6b3a7e9ced2b/volumes" Nov 03 20:55:52.891852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.891810 5932 generic.go:356] "Generic (PLEG): container finished" podID="2145eea5-0bc2-4812-9538-b6325b622e38" containerID="70825b91580b2f3822fafe3e164e40e4e9687573059ec00cb6d129717e3bc683" exitCode=0 Nov 03 20:55:52.892380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.891949 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" event={"ID":"2145eea5-0bc2-4812-9538-b6325b622e38","Type":"ContainerDied","Data":"70825b91580b2f3822fafe3e164e40e4e9687573059ec00cb6d129717e3bc683"} Nov 03 20:55:52.892380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.892040 5932 scope.go:122] "RemoveContainer" containerID="b4daa8080a00865c328a1ec4351f965060499399fd2601731866b5347480e86a" Nov 03 20:55:52.894985 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.894960 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:52.895997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:52.895974 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:55:53.074556 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.074516 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:55:53.148713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.148546 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.148713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.148594 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.148713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.148613 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.148998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.148791 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.148998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.148946 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.148998 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.148989 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.149161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149050 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.149161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149111 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.149161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149141 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy\") pod \"2145eea5-0bc2-4812-9538-b6325b622e38\" (UID: \"2145eea5-0bc2-4812-9538-b6325b622e38\") " Nov 03 20:55:53.149523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149496 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:55:53.149523 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149505 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:53.149763 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149739 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:53.149831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149787 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:53.149922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.149906 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:53.150485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.150462 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.150588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.150487 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2145eea5-0bc2-4812-9538-b6325b622e38-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.150588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.150502 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.150588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.150514 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.150588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.150527 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.153312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.153280 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:55:53.153456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.153435 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:53.154140 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.154113 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "kube-api-access-kgnbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:55:53.155018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.154993 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo" pod "2145eea5-0bc2-4812-9538-b6325b622e38" (UID: "2145eea5-0bc2-4812-9538-b6325b622e38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:55:53.252591 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.252554 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2145eea5-0bc2-4812-9538-b6325b622e38-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.252591 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.252591 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.252812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.252606 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2145eea5-0bc2-4812-9538-b6325b622e38-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.252812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.252634 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgnbm\" (UniqueName: \"kubernetes.io/projected/2145eea5-0bc2-4812-9538-b6325b622e38-kube-api-access-kgnbm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:55:53.987777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.987723 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" event={"ID":"2145eea5-0bc2-4812-9538-b6325b622e38","Type":"ContainerDied","Data":"b9f2d1ac71bc77582ca4c32d0892a7d10a4436659e38ab2d2038dd359e0da4f6"} Nov 03 20:55:53.988266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.987820 5932 scope.go:122] "RemoveContainer" containerID="70825b91580b2f3822fafe3e164e40e4e9687573059ec00cb6d129717e3bc683" Nov 03 20:55:53.988606 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:53.988584 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2" Nov 03 20:55:54.163647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.162675 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf"] Nov 03 20:55:54.244857 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.238681 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae926a78-kbtpf"] Nov 03 20:55:54.297201 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.297094 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2"] Nov 03 20:55:54.469980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.469774 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-15564eb1-openshift-default-77f65d6bff-zght2"] Nov 03 20:55:54.500838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.478027 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv"] Nov 03 20:55:54.500838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.491662 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f9337cba-jmhpv"] Nov 03 20:55:54.527660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.504739 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh"] Nov 03 20:55:54.527660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:54.504993 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" containerID="cri-o://14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1" gracePeriod=30 Nov 03 20:55:56.054094 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:56.054062 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02b969eb-b724-4f53-8685-507c1c1fc9c9" path="/var/lib/kubelet/pods/02b969eb-b724-4f53-8685-507c1c1fc9c9/volumes" Nov 03 20:55:56.054744 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:56.054727 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" path="/var/lib/kubelet/pods/2145eea5-0bc2-4812-9538-b6325b622e38/volumes" Nov 03 20:55:56.055274 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:56.055260 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b1d972-1316-4093-8f3a-d090d52992e9" path="/var/lib/kubelet/pods/c3b1d972-1316-4093-8f3a-d090d52992e9/volumes" Nov 03 20:55:57.152643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.152519 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq"] Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154726 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154754 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154797 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154805 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154900 5932 cpu_manager.go:414] "containerMap: removing container" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154909 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154937 5932 cpu_manager.go:414] "containerMap: removing container" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" Nov 03 20:55:57.155141 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.154958 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" Nov 03 20:55:57.155697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.155437 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" Nov 03 20:55:57.155697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.155480 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" Nov 03 20:55:57.155697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.155501 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2145eea5-0bc2-4812-9538-b6325b622e38" containerName="istio-proxy" Nov 03 20:55:57.477405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.471697 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq"] Nov 03 20:55:57.477405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.476432 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f"] Nov 03 20:55:57.477405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.475785 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.487123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.483415 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="21be1d12-e5f0-4161-a549-6b3a7e9ced2b" containerName="istio-proxy" Nov 03 20:55:57.498583 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.498541 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f"] Nov 03 20:55:57.498770 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.498715 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.539076 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.539041 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-df4313ad\"" Nov 03 20:55:57.585328 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.585302 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-df4313ad-dockercfg-vgdhq\"" Nov 03 20:55:57.590303 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.590279 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.590522 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.590505 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njqcb\" (UniqueName: \"kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb\") pod \"payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.590691 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.590660 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume\") pod \"payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.590767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.590705 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jhz\" (UniqueName: \"kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz\") pod \"payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.693071 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.693034 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njqcb\" (UniqueName: \"kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb\") pod \"payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.693268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.693138 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume\") pod \"payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.693268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.693156 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jhz\" (UniqueName: \"kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz\") pod \"payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.693268 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.693249 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.694155 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.694129 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume\") pod \"payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.694214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.694164 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.751524 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.751407 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jhz\" (UniqueName: \"kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz\") pod \"payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:57.751748 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.751588 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njqcb\" (UniqueName: \"kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb\") pod \"payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.808717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.808689 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:57.816892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:57.816865 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:58.196633 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:58.196579 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq"] Nov 03 20:55:58.238264 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:55:58.238222 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937486ff_920a_492b_8e55_e9cf2cbf14e5.slice/crio-8eab87670175b68e17cebce6bb12d1e6b68500a98a4b5b576f34901fb8c68917 WatchSource:0}: Error finding container 8eab87670175b68e17cebce6bb12d1e6b68500a98a4b5b576f34901fb8c68917: Status 404 returned error can't find the container with id 8eab87670175b68e17cebce6bb12d1e6b68500a98a4b5b576f34901fb8c68917 Nov 03 20:55:58.249314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:58.249288 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f"] Nov 03 20:55:58.254008 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:55:58.253982 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca2e14f_72f0_428c_b56e_d10850407dfc.slice/crio-9abb55afaab4b74dc055eda23064db8baee3461407268f0da79b9e342772f38b WatchSource:0}: Error finding container 9abb55afaab4b74dc055eda23064db8baee3461407268f0da79b9e342772f38b: Status 404 returned error can't find the container with id 9abb55afaab4b74dc055eda23064db8baee3461407268f0da79b9e342772f38b Nov 03 20:55:58.257090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:58.257062 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" event={"ID":"937486ff-920a-492b-8e55-e9cf2cbf14e5","Type":"ContainerStarted","Data":"8eab87670175b68e17cebce6bb12d1e6b68500a98a4b5b576f34901fb8c68917"} Nov 03 20:55:59.268183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.268144 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" event={"ID":"937486ff-920a-492b-8e55-e9cf2cbf14e5","Type":"ContainerStarted","Data":"b780f695e5009bfd440b049d1acb6c0b6d1f7883005e3b18ef444c87c5b716d5"} Nov 03 20:55:59.268596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.268217 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:55:59.270024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.269995 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" event={"ID":"1ca2e14f-72f0-428c-b56e-d10850407dfc","Type":"ContainerStarted","Data":"ae1880dfadb911cc8e118c9ec2244ef8feb660be50f1c4da8547273c26aeb898"} Nov 03 20:55:59.270126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.270038 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" event={"ID":"1ca2e14f-72f0-428c-b56e-d10850407dfc","Type":"ContainerStarted","Data":"9abb55afaab4b74dc055eda23064db8baee3461407268f0da79b9e342772f38b"} Nov 03 20:55:59.270126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.270058 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:55:59.384061 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.383998 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" podStartSLOduration=3.383984279 podStartE2EDuration="3.383984279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:59.380550681 +0000 UTC m=+1567.445792154" watchObservedRunningTime="2026-11-03 20:55:59.383984279 +0000 UTC m=+1567.449225770" Nov 03 20:55:59.514653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.511033 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" podStartSLOduration=3.511008743 podStartE2EDuration="3.511008743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:55:59.488760303 +0000 UTC m=+1567.554001777" watchObservedRunningTime="2026-11-03 20:55:59.511008743 +0000 UTC m=+1567.576250217" Nov 03 20:55:59.970119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:55:59.970090 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:56:00.046789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.046760 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.046959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.046843 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.046959 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.046935 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047068 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.046993 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047086 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:00.047180 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047121 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047244 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047369 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047393 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:56:00.047527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047405 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047468 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs\") pod \"917665c3-87ad-4ad0-b662-8358ea49e3a1\" (UID: \"917665c3-87ad-4ad0-b662-8358ea49e3a1\") " Nov 03 20:56:00.047779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047642 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:00.047779 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047757 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:00.047982 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.047951 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:00.048319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.048297 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.048431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.048323 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.048431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.048343 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.048431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.048358 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/917665c3-87ad-4ad0-b662-8358ea49e3a1-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.048431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.048372 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.049528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.049500 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "kube-api-access-v9z4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:00.049617 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.049525 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:56:00.049912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.049889 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:56:00.049973 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.049959 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token" pod "917665c3-87ad-4ad0-b662-8358ea49e3a1" (UID: "917665c3-87ad-4ad0-b662-8358ea49e3a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:56:00.150912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.150875 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9z4z\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-kube-api-access-v9z4z\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.150912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.150908 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.150912 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.150921 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.151188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.150929 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/917665c3-87ad-4ad0-b662-8358ea49e3a1-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:56:00.281464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.281378 5932 generic.go:356] "Generic (PLEG): container finished" podID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerID="14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1" exitCode=0 Nov 03 20:56:00.281464 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.281432 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" Nov 03 20:56:00.282090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.281457 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" event={"ID":"917665c3-87ad-4ad0-b662-8358ea49e3a1","Type":"ContainerDied","Data":"14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1"} Nov 03 20:56:00.282090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.281512 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh" event={"ID":"917665c3-87ad-4ad0-b662-8358ea49e3a1","Type":"ContainerDied","Data":"32ffec728fadf2769cc4339cc14721431de05fa29d88a44d24b00af6f06b4241"} Nov 03 20:56:00.282090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.281535 5932 scope.go:122] "RemoveContainer" containerID="14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1" Nov 03 20:56:00.294183 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.294164 5932 scope.go:122] "RemoveContainer" containerID="ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8" Nov 03 20:56:00.304713 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.304691 5932 scope.go:122] "RemoveContainer" containerID="14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1" Nov 03 20:56:00.304990 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:56:00.304968 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1\": container with ID starting with 14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1 not found: ID does not exist" containerID="14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1" Nov 03 20:56:00.305057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.304997 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1"} err="failed to get container status \"14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1\": rpc error: code = NotFound desc = could not find container \"14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1\": container with ID starting with 14c17991f5086e13f4de436cbe379612d047a94f304475b1f9c99cc64849f7c1 not found: ID does not exist" Nov 03 20:56:00.305057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.305014 5932 scope.go:122] "RemoveContainer" containerID="ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8" Nov 03 20:56:00.305242 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:56:00.305223 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8\": container with ID starting with ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8 not found: ID does not exist" containerID="ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8" Nov 03 20:56:00.305322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.305245 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8"} err="failed to get container status \"ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8\": rpc error: code = NotFound desc = could not find container \"ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8\": container with ID starting with ff978c65353a5ac6f1b37782971b221ea256e52fcfbe7bbd9cd6175be51861a8 not found: ID does not exist" Nov 03 20:56:00.450679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.450609 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh"] Nov 03 20:56:00.453096 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:00.453071 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f9337cba-openshift-default-55ff6bb7b5-w8rhh"] Nov 03 20:56:01.867375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.867344 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv"] Nov 03 20:56:01.868190 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.868174 5932 cpu_manager.go:414] "containerMap: removing container" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" Nov 03 20:56:01.868233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.868192 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" Nov 03 20:56:01.868233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.868211 5932 cpu_manager.go:414] "containerMap: removing container" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" Nov 03 20:56:01.868233 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.868216 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" Nov 03 20:56:01.868358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.868348 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" Nov 03 20:56:01.873949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.873929 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:01.904429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.904399 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j8nv\" (UniqueName: \"kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv\") pod \"payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:01.904567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:01.904553 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:02.015131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.015098 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-299a741b\"" Nov 03 20:56:02.023686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.019701 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j8nv\" (UniqueName: \"kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv\") pod \"payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:02.023686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.019905 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:02.051829 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.051794 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-299a741b-dockercfg-m9tsk\"" Nov 03 20:56:02.146655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.132540 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:02.287655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.281916 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" path="/var/lib/kubelet/pods/917665c3-87ad-4ad0-b662-8358ea49e3a1/volumes" Nov 03 20:56:02.369988 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.369959 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv"] Nov 03 20:56:02.396140 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.392414 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j8nv\" (UniqueName: \"kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv\") pod \"payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:02.396140 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.392487 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k"] Nov 03 20:56:02.397287 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.396775 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="917665c3-87ad-4ad0-b662-8358ea49e3a1" containerName="istio-proxy" Nov 03 20:56:02.418676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.418597 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k"] Nov 03 20:56:02.418969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.418952 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.489340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.489297 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:02.553378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.553352 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fvrl\" (UniqueName: \"kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl\") pod \"payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.553488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.553444 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume\") pod \"payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.655003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.654920 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fvrl\" (UniqueName: \"kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl\") pod \"payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.655003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.654983 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume\") pod \"payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.655836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.655815 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume\") pod \"payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.713173 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.713143 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fvrl\" (UniqueName: \"kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl\") pod \"payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.736471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.736448 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:02.876863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:02.876834 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv"] Nov 03 20:56:02.879951 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:56:02.879920 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160e8882_42ee_46ee_8df9_336323a0b1f0.slice/crio-8a301e3c2318ce207a8db49311be81c0c39d7bc965aa0e8d0938ef71ec06ea9f WatchSource:0}: Error finding container 8a301e3c2318ce207a8db49311be81c0c39d7bc965aa0e8d0938ef71ec06ea9f: Status 404 returned error can't find the container with id 8a301e3c2318ce207a8db49311be81c0c39d7bc965aa0e8d0938ef71ec06ea9f Nov 03 20:56:03.124126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.124101 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k"] Nov 03 20:56:03.126973 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:56:03.126946 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77275cb9_5cde_42eb_ba3b_b842aa392759.slice/crio-c6b34a33e50c0a17956d05e86f6121f8a2171d4e2876c7567799910c3990002d WatchSource:0}: Error finding container c6b34a33e50c0a17956d05e86f6121f8a2171d4e2876c7567799910c3990002d: Status 404 returned error can't find the container with id c6b34a33e50c0a17956d05e86f6121f8a2171d4e2876c7567799910c3990002d Nov 03 20:56:03.381954 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.381918 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" event={"ID":"160e8882-42ee-46ee-8df9-336323a0b1f0","Type":"ContainerStarted","Data":"6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615"} Nov 03 20:56:03.382114 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.381971 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" event={"ID":"160e8882-42ee-46ee-8df9-336323a0b1f0","Type":"ContainerStarted","Data":"8a301e3c2318ce207a8db49311be81c0c39d7bc965aa0e8d0938ef71ec06ea9f"} Nov 03 20:56:03.382114 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.382004 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:03.384846 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.384819 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" event={"ID":"77275cb9-5cde-42eb-ba3b-b842aa392759","Type":"ContainerStarted","Data":"a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33"} Nov 03 20:56:03.384975 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.384860 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" event={"ID":"77275cb9-5cde-42eb-ba3b-b842aa392759","Type":"ContainerStarted","Data":"c6b34a33e50c0a17956d05e86f6121f8a2171d4e2876c7567799910c3990002d"} Nov 03 20:56:03.384975 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.384915 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:03.495680 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.495559 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" podStartSLOduration=2.495545366 podStartE2EDuration="2.495545366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:03.489635359 +0000 UTC m=+1571.554876829" watchObservedRunningTime="2026-11-03 20:56:03.495545366 +0000 UTC m=+1571.560786840" Nov 03 20:56:03.601280 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:03.601219 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" podStartSLOduration=2.601200709 podStartE2EDuration="2.601200709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:03.597739342 +0000 UTC m=+1571.662980818" watchObservedRunningTime="2026-11-03 20:56:03.601200709 +0000 UTC m=+1571.666442185" Nov 03 20:56:10.285157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:10.285127 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:56:10.285661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:10.285197 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:56:12.059842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:12.059808 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl"] Nov 03 20:56:12.095518 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:12.095487 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bc92735-kb9zl"] Nov 03 20:56:14.054500 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:14.054467 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0faac424-a52e-46d1-9c66-f06f58b5c421" path="/var/lib/kubelet/pods/0faac424-a52e-46d1-9c66-f06f58b5c421/volumes" Nov 03 20:56:14.153132 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:56:14.153097 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:56:14.397478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:14.397452 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:56:14.397658 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:14.397532 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:56:18.054942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:18.054909 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6"] Nov 03 20:56:18.094311 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:18.094280 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8234aaa-d26c6"] Nov 03 20:56:20.054289 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:20.054256 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1081749c-9383-4400-871d-e47f523adb92" path="/var/lib/kubelet/pods/1081749c-9383-4400-871d-e47f523adb92/volumes" Nov 03 20:56:36.054314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:36.054281 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht"] Nov 03 20:56:36.093161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:36.093131 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4753bf9b-vtnht"] Nov 03 20:56:38.066972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:38.066935 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af506dab-2d8d-4dc6-aa76-0f77378601c3" path="/var/lib/kubelet/pods/af506dab-2d8d-4dc6-aa76-0f77378601c3/volumes" Nov 03 20:56:39.060829 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:39.060779 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt"] Nov 03 20:56:39.099115 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:39.099079 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-594ce3d0-mxhxt"] Nov 03 20:56:40.112147 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:40.112114 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1129f80-eb89-44d4-8f10-f8e302b6f37a" path="/var/lib/kubelet/pods/d1129f80-eb89-44d4-8f10-f8e302b6f37a/volumes" Nov 03 20:56:44.151658 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:56:44.151588 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:56:44.833649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.827156 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr"] Nov 03 20:56:44.851901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.851867 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr"] Nov 03 20:56:44.852465 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.852378 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.925719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.915081 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-cc5d0f-openshift-default-dockercfg-v86ns\"" Nov 03 20:56:44.989471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.989431 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.989979 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.989938 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.990399 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.990377 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.998650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.994837 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.998650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.994912 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.998650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.995031 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.998650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.995058 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.998650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.995099 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hct9c\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:44.998650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:44.995195 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142116 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142208 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142267 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142374 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142403 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142442 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hct9c\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142535 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142670 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.142783 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.143583 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.143888 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.145681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.144992 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.147828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.145799 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.147828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.147240 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.151285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.150829 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.151285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.151253 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.200942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.200913 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.202436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.202385 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hct9c\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c\") pod \"e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.451120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.451036 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:45.904757 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.903840 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr"] Nov 03 20:56:45.908720 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:56:45.908694 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2314c3a7_3dbe_4574_9c32_42175696ec7c.slice/crio-51ab4b9f4302bdec341717a35cc26a9e4bbd55665df72a3756e8af77ececd9cb WatchSource:0}: Error finding container 51ab4b9f4302bdec341717a35cc26a9e4bbd55665df72a3756e8af77ececd9cb: Status 404 returned error can't find the container with id 51ab4b9f4302bdec341717a35cc26a9e4bbd55665df72a3756e8af77ececd9cb Nov 03 20:56:45.927324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.925677 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:56:45.927324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.925781 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:56:45.927324 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:45.925823 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:56:46.172110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:46.172029 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" event={"ID":"2314c3a7-3dbe-4574-9c32-42175696ec7c","Type":"ContainerStarted","Data":"66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387"} Nov 03 20:56:46.172110 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:46.172096 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" event={"ID":"2314c3a7-3dbe-4574-9c32-42175696ec7c","Type":"ContainerStarted","Data":"51ab4b9f4302bdec341717a35cc26a9e4bbd55665df72a3756e8af77ececd9cb"} Nov 03 20:56:46.288117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:46.288054 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podStartSLOduration=2.288039667 podStartE2EDuration="2.288039667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:46.284442677 +0000 UTC m=+1614.349684150" watchObservedRunningTime="2026-11-03 20:56:46.288039667 +0000 UTC m=+1614.353281141" Nov 03 20:56:46.453699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:46.453473 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:47.457045 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:47.457016 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:48.055372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:48.055343 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm"] Nov 03 20:56:48.055812 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:48.055793 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9657716f-srwgm"] Nov 03 20:56:48.191285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:48.191254 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:48.192312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:48.192291 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:56:50.054477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:50.054437 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac6017bb-c0b7-43cf-b8e7-7678d3f1017c" path="/var/lib/kubelet/pods/ac6017bb-c0b7-43cf-b8e7-7678d3f1017c/volumes" Nov 03 20:56:51.049896 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:51.049868 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck"] Nov 03 20:56:51.095987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:51.095956 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00507ce5-dxjck"] Nov 03 20:56:52.054654 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:52.054578 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4" path="/var/lib/kubelet/pods/a84cb05e-0fb3-4c4b-b9a1-10428ad5d3a4/volumes" Nov 03 20:56:52.900697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:52.900353 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq"] Nov 03 20:56:53.028701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.020986 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.074054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.074013 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-cc5d0f\"" Nov 03 20:56:53.146668 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.143667 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-cc5d0f-dockercfg-r2nr8\"" Nov 03 20:56:53.214001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.209017 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbt9d\" (UniqueName: \"kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d\") pod \"payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.214001 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.209160 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.330732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.330687 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbt9d\" (UniqueName: \"kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d\") pod \"payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.331109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.331087 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.332831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.332805 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.438778 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.438162 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq"] Nov 03 20:56:53.463018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.455615 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd"] Nov 03 20:56:53.518041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.518008 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbt9d\" (UniqueName: \"kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d\") pod \"payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.545013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.544975 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd"] Nov 03 20:56:53.545244 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.545113 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.545358 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.545339 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:53.659154 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.659097 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxk9\" (UniqueName: \"kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9\") pod \"payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.659312 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.659216 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.761339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.761245 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxk9\" (UniqueName: \"kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9\") pod \"payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.761518 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.761344 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.762127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.762105 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.819127 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.819096 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxk9\" (UniqueName: \"kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9\") pod \"payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.875714 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.875686 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:53.957948 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:53.957917 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq"] Nov 03 20:56:53.964675 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:56:53.960922 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63ff093_d891_4d35_8881_f24ae99fc54c.slice/crio-a12f6fe6d11e469d53a04c2ae7212dba4f9ce3faa1cc00fce3c675c8fd2b73bc WatchSource:0}: Error finding container a12f6fe6d11e469d53a04c2ae7212dba4f9ce3faa1cc00fce3c675c8fd2b73bc: Status 404 returned error can't find the container with id a12f6fe6d11e469d53a04c2ae7212dba4f9ce3faa1cc00fce3c675c8fd2b73bc Nov 03 20:56:54.266021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.265791 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd"] Nov 03 20:56:54.269060 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:56:54.268990 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd320fa_0802_4b5a_8c9f_32cd9f43b59b.slice/crio-db58b94152da7e583d89b04e1e68a28b2cebefa29285b44b9e1b80091c2dd575 WatchSource:0}: Error finding container db58b94152da7e583d89b04e1e68a28b2cebefa29285b44b9e1b80091c2dd575: Status 404 returned error can't find the container with id db58b94152da7e583d89b04e1e68a28b2cebefa29285b44b9e1b80091c2dd575 Nov 03 20:56:54.516841 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.516803 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" event={"ID":"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b","Type":"ContainerStarted","Data":"e6a93140c37ff36b18d99ebe8a9fff40b408939a7a225e6ff5c1a6fc765ce3d4"} Nov 03 20:56:54.517024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.516871 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" event={"ID":"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b","Type":"ContainerStarted","Data":"db58b94152da7e583d89b04e1e68a28b2cebefa29285b44b9e1b80091c2dd575"} Nov 03 20:56:54.517024 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.516905 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:56:54.518911 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.518883 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" event={"ID":"c63ff093-d891-4d35-8881-f24ae99fc54c","Type":"ContainerStarted","Data":"09c4691bfe610da30510188bd3996d32ffb7d4a24f34256aab195a3ee53a8d06"} Nov 03 20:56:54.519012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.518933 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" event={"ID":"c63ff093-d891-4d35-8881-f24ae99fc54c","Type":"ContainerStarted","Data":"a12f6fe6d11e469d53a04c2ae7212dba4f9ce3faa1cc00fce3c675c8fd2b73bc"} Nov 03 20:56:54.519012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.518960 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:56:54.627162 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.627090 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" podStartSLOduration=2.627071298 podStartE2EDuration="2.627071298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:54.624871396 +0000 UTC m=+1622.690112870" watchObservedRunningTime="2026-11-03 20:56:54.627071298 +0000 UTC m=+1622.692312773" Nov 03 20:56:54.735387 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.735326 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" podStartSLOduration=2.735307014 podStartE2EDuration="2.735307014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:56:54.731921881 +0000 UTC m=+1622.797163355" watchObservedRunningTime="2026-11-03 20:56:54.735307014 +0000 UTC m=+1622.800548489" Nov 03 20:56:54.934728 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:54.930726 5932 scope.go:122] "RemoveContainer" containerID="022d5350fc5654ce2564df7c40c557d3b4398df329ea03477ef70b7f51a5e518" Nov 03 20:56:55.000801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:55.000769 5932 scope.go:122] "RemoveContainer" containerID="6488878d3b9a48aa4e7e960b854cf947776688f44c29c52506353d44f780ce07" Nov 03 20:56:55.046558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:56:55.046534 5932 scope.go:122] "RemoveContainer" containerID="bd7e377306175fa1bc658950c7c85af0cc8f6d985513d2b1c4ca9f6018d0dcee" Nov 03 20:57:01.054350 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:01.054318 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684"] Nov 03 20:57:01.099209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:01.099180 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d26abf-55684"] Nov 03 20:57:02.068527 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:02.068495 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5baad81-1a62-41d3-a967-be26eae51b29" path="/var/lib/kubelet/pods/a5baad81-1a62-41d3-a967-be26eae51b29/volumes" Nov 03 20:57:05.553501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.553472 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 20:57:05.566852 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.566824 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 20:57:05.574565 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.574525 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll"] Nov 03 20:57:05.583126 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.583099 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.624199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.624129 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll"] Nov 03 20:57:05.632077 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.632053 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:05.649398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.649369 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:57:05.672887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.672845 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjg87\" (UniqueName: \"kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.673724 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.673700 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.674055 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.674035 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.776420 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.776383 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjg87\" (UniqueName: \"kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.776643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.776469 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.776643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.776584 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.777662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.777584 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.779313 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.779289 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.835184 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.835147 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjg87\" (UniqueName: \"kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87\") pod \"maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:05.907176 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:05.907135 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:06.061928 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:06.061899 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk"] Nov 03 20:57:06.062093 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:06.061938 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a4ebd53-hpfpk"] Nov 03 20:57:06.304187 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:06.303964 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll"] Nov 03 20:57:06.308504 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:57:06.308471 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68913a47_fec4_4dce_9ab2_eb94f35ac138.slice/crio-5e9253941c645711adb54b79417e7a75d85f601c93d85ea897efe3da9f379675 WatchSource:0}: Error finding container 5e9253941c645711adb54b79417e7a75d85f601c93d85ea897efe3da9f379675: Status 404 returned error can't find the container with id 5e9253941c645711adb54b79417e7a75d85f601c93d85ea897efe3da9f379675 Nov 03 20:57:06.709432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:06.709391 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" event={"ID":"68913a47-fec4-4dce-9ab2-eb94f35ac138","Type":"ContainerStarted","Data":"5e9253941c645711adb54b79417e7a75d85f601c93d85ea897efe3da9f379675"} Nov 03 20:57:07.722338 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:07.722292 5932 generic.go:356] "Generic (PLEG): container finished" podID="68913a47-fec4-4dce-9ab2-eb94f35ac138" containerID="cddce299fa214f7f2725e901733c12eebcbab3f9fdca34f262b7b49e5ed79d37" exitCode=0 Nov 03 20:57:07.722828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:07.722383 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" event={"ID":"68913a47-fec4-4dce-9ab2-eb94f35ac138","Type":"ContainerDied","Data":"cddce299fa214f7f2725e901733c12eebcbab3f9fdca34f262b7b49e5ed79d37"} Nov 03 20:57:08.058882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:08.058804 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a101eb8-9bd7-4410-9500-91db0893592c" path="/var/lib/kubelet/pods/2a101eb8-9bd7-4410-9500-91db0893592c/volumes" Nov 03 20:57:08.102424 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:08.102388 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr"] Nov 03 20:57:08.107854 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:08.107826 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18e6420a-9jmmr"] Nov 03 20:57:08.928372 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:08.928338 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:08.935851 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:08.935809 5932 patch_prober.go:29] interesting pod/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:57:08.935988 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:08.935878 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:57:09.030271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.030194 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca\") pod \"68913a47-fec4-4dce-9ab2-eb94f35ac138\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " Nov 03 20:57:09.030271 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.030240 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87\" (UniqueName: \"kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87\") pod \"68913a47-fec4-4dce-9ab2-eb94f35ac138\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " Nov 03 20:57:09.030480 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.030425 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca\") pod \"68913a47-fec4-4dce-9ab2-eb94f35ac138\" (UID: \"68913a47-fec4-4dce-9ab2-eb94f35ac138\") " Nov 03 20:57:09.030618 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.030589 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca" pod "68913a47-fec4-4dce-9ab2-eb94f35ac138" (UID: "68913a47-fec4-4dce-9ab2-eb94f35ac138"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:09.031066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.031041 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:09.032518 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.032490 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca" pod "68913a47-fec4-4dce-9ab2-eb94f35ac138" (UID: "68913a47-fec4-4dce-9ab2-eb94f35ac138"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:09.032666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.032646 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87" pod "68913a47-fec4-4dce-9ab2-eb94f35ac138" (UID: "68913a47-fec4-4dce-9ab2-eb94f35ac138"). InnerVolumeSpecName "kube-api-access-rjg87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:09.134451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.133235 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68913a47-fec4-4dce-9ab2-eb94f35ac138-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:09.134451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.133261 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjg87\" (UniqueName: \"kubernetes.io/projected/68913a47-fec4-4dce-9ab2-eb94f35ac138-kube-api-access-rjg87\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:09.745568 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.745535 5932 generic.go:356] "Generic (PLEG): container finished" podID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerID="9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb" exitCode=0 Nov 03 20:57:09.745777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.745605 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" event={"ID":"94be4b6a-4a95-40e7-9ead-b4f02fd232c5","Type":"ContainerDied","Data":"9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb"} Nov 03 20:57:09.746083 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.746066 5932 scope.go:122] "RemoveContainer" containerID="9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb" Nov 03 20:57:09.747335 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.747318 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" Nov 03 20:57:09.747448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.747357 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll" event={"ID":"68913a47-fec4-4dce-9ab2-eb94f35ac138","Type":"ContainerDied","Data":"5e9253941c645711adb54b79417e7a75d85f601c93d85ea897efe3da9f379675"} Nov 03 20:57:09.747448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.747419 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e9253941c645711adb54b79417e7a75d85f601c93d85ea897efe3da9f379675" Nov 03 20:57:09.747661 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.747634 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:09.747733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.747700 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:09.747733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:09.747731 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:10.054940 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:10.054852 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e82ac3-ba3c-4789-a24f-c413746c0d14" path="/var/lib/kubelet/pods/99e82ac3-ba3c-4789-a24f-c413746c0d14/volumes" Nov 03 20:57:10.762943 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:10.762909 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" event={"ID":"94be4b6a-4a95-40e7-9ead-b4f02fd232c5","Type":"ContainerStarted","Data":"21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d"} Nov 03 20:57:11.732348 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:11.732316 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:57:11.733204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:11.732941 5932 patch_prober.go:29] interesting pod/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:57:11.733392 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:11.733367 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 20:57:12.745646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:12.745215 5932 patch_prober.go:29] interesting pod/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:57:12.745646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:12.745278 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 20:57:12.952899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:12.952865 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l"] Nov 03 20:57:12.953075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:12.953052 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" containerID="cri-o://b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8" gracePeriod=30 Nov 03 20:57:13.729951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:13.729912 5932 patch_prober.go:29] interesting pod/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:57:13.730248 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:13.730218 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 20:57:14.175605 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:57:14.175569 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:57:14.741645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:14.731926 5932 patch_prober.go:29] interesting pod/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:57:14.741645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:14.731984 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 20:57:15.734508 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:15.734467 5932 patch_prober.go:29] interesting pod/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:57:15.735148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:15.735112 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 20:57:16.733286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:16.733251 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:57:16.733787 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:16.733749 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:57:16.735323 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:16.735299 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:57:17.103234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:17.103156 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57"] Nov 03 20:57:17.239539 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:17.239511 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c"] Nov 03 20:57:18.054872 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.054840 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj"] Nov 03 20:57:18.056693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.056663 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b15f3f-hhqfj"] Nov 03 20:57:18.251817 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.251787 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:57:18.426277 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426246 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426289 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426336 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426446 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426390 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426564 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426545 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426598 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426660 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426677 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:18.426776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426730 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426789 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy\") pod \"96e07592-beae-4eb9-a59b-500ed6d90744\" (UID: \"96e07592-beae-4eb9-a59b-500ed6d90744\") " Nov 03 20:57:18.426927 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.426901 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:18.427185 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427129 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427055 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427410 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427812 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427834 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427850 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427864 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.427903 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.427877 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96e07592-beae-4eb9-a59b-500ed6d90744-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.429731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.429572 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:18.429731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.429667 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:18.429731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.429689 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:18.430382 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.430359 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj" pod "96e07592-beae-4eb9-a59b-500ed6d90744" (UID: "96e07592-beae-4eb9-a59b-500ed6d90744"). InnerVolumeSpecName "kube-api-access-nwdkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:18.529258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.529178 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.529258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.529205 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96e07592-beae-4eb9-a59b-500ed6d90744-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.529258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.529216 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96e07592-beae-4eb9-a59b-500ed6d90744-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:18.529258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:18.529226 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwdkj\" (UniqueName: \"kubernetes.io/projected/96e07592-beae-4eb9-a59b-500ed6d90744-kube-api-access-nwdkj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:19.039412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.039366 5932 generic.go:356] "Generic (PLEG): container finished" podID="96e07592-beae-4eb9-a59b-500ed6d90744" containerID="b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8" exitCode=0 Nov 03 20:57:19.039753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.039722 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" event={"ID":"96e07592-beae-4eb9-a59b-500ed6d90744","Type":"ContainerDied","Data":"b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8"} Nov 03 20:57:19.040003 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.039985 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" event={"ID":"96e07592-beae-4eb9-a59b-500ed6d90744","Type":"ContainerDied","Data":"e9f8029de220b108a09da105e6c994774d8931e04a12c539d1829215fb3b50f6"} Nov 03 20:57:19.040138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.040124 5932 scope.go:122] "RemoveContainer" containerID="b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8" Nov 03 20:57:19.040403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.039847 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l" Nov 03 20:57:19.162613 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.162584 5932 scope.go:122] "RemoveContainer" containerID="b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8" Nov 03 20:57:19.183760 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:57:19.183593 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8\": container with ID starting with b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8 not found: ID does not exist" containerID="b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8" Nov 03 20:57:19.183760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.183659 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8"} err="failed to get container status \"b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8\": rpc error: code = NotFound desc = could not find container \"b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8\": container with ID starting with b718018f2ae01d7f6eae2b7d19453c3b6a0b5a171e9c2e623f8cf488918384b8 not found: ID does not exist" Nov 03 20:57:19.304876 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.302724 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l"] Nov 03 20:57:19.327930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:19.327884 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-sqf2l"] Nov 03 20:57:20.054551 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:20.054518 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" path="/var/lib/kubelet/pods/96e07592-beae-4eb9-a59b-500ed6d90744/volumes" Nov 03 20:57:20.055109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:20.055090 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c015ab1b-0a13-4ca9-85c4-b938cc71cd93" path="/var/lib/kubelet/pods/c015ab1b-0a13-4ca9-85c4-b938cc71cd93/volumes" Nov 03 20:57:21.062190 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.062155 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h"] Nov 03 20:57:21.063202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.063178 5932 cpu_manager.go:414] "containerMap: removing container" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" Nov 03 20:57:21.063202 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.063202 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" Nov 03 20:57:21.063380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.063262 5932 cpu_manager.go:414] "containerMap: removing container" podUID="68913a47-fec4-4dce-9ab2-eb94f35ac138" containerName="revoke-keys" Nov 03 20:57:21.063380 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.063271 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68913a47-fec4-4dce-9ab2-eb94f35ac138" containerName="revoke-keys" Nov 03 20:57:21.063543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.063528 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e07592-beae-4eb9-a59b-500ed6d90744" containerName="istio-proxy" Nov 03 20:57:21.063605 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.063555 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="68913a47-fec4-4dce-9ab2-eb94f35ac138" containerName="revoke-keys" Nov 03 20:57:21.069247 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.069225 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.123917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.123881 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e68f8112-openshift-default-dockercfg-cp47h\"" Nov 03 20:57:21.158012 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.157948 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.158447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.158423 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh8s\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.158535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.158476 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.158754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.158736 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.158853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.158827 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.158898 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.158853 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.158961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.158947 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.159173 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.159159 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.159274 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.159252 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.172046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.172012 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h"] Nov 03 20:57:21.299949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.266724 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8s\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.300296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.300267 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.313753 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.300550 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.495648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.314261 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.495648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.393966 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh8s\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.495648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.397352 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.496786 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.496825 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.496908 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.497096 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.497178 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.497253 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.498214 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.498498 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.504945 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.505193 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.545667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.534322 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.564695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.561787 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:21.686084 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:21.686047 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:22.055581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.055549 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z"] Nov 03 20:57:22.055977 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.055964 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr"] Nov 03 20:57:22.060742 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.060722 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60365f29-r5fpr"] Nov 03 20:57:22.065393 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.065368 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f66e21fa-nzl6z"] Nov 03 20:57:22.104079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.104033 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" podUID="da90ae73-1008-4edd-9b3e-c4011c77a46f" containerName="payload-processing" containerID="cri-o://77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb" gracePeriod=25 Nov 03 20:57:22.217088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.217062 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h"] Nov 03 20:57:22.219930 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:57:22.219904 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d0cb91_0699_494a_9595_7d06a9c49e60.slice/crio-f730679be0b1542705f0edb00698656d2f622de1daa8fee3b78cbddc154645a9 WatchSource:0}: Error finding container f730679be0b1542705f0edb00698656d2f622de1daa8fee3b78cbddc154645a9: Status 404 returned error can't find the container with id f730679be0b1542705f0edb00698656d2f622de1daa8fee3b78cbddc154645a9 Nov 03 20:57:22.222357 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.222322 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:22.222478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.222402 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:22.222478 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.222449 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:22.240600 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.240566 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" podUID="5b250057-e693-476f-b75f-afb439b384a8" containerName="payload-pre-processing" containerID="cri-o://5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1" gracePeriod=25 Nov 03 20:57:22.439751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.439722 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:57:22.606054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.606026 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:57:22.631884 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.631846 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2\" (UniqueName: \"kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2\") pod \"da90ae73-1008-4edd-9b3e-c4011c77a46f\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " Nov 03 20:57:22.632066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.631940 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume\") pod \"da90ae73-1008-4edd-9b3e-c4011c77a46f\" (UID: \"da90ae73-1008-4edd-9b3e-c4011c77a46f\") " Nov 03 20:57:22.632340 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.632311 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume" pod "da90ae73-1008-4edd-9b3e-c4011c77a46f" (UID: "da90ae73-1008-4edd-9b3e-c4011c77a46f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:22.632755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.632733 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da90ae73-1008-4edd-9b3e-c4011c77a46f-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:22.634266 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.634242 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2" pod "da90ae73-1008-4edd-9b3e-c4011c77a46f" (UID: "da90ae73-1008-4edd-9b3e-c4011c77a46f"). InnerVolumeSpecName "kube-api-access-5mds2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:22.733577 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.733525 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume\") pod \"5b250057-e693-476f-b75f-afb439b384a8\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " Nov 03 20:57:22.733980 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.733955 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg\" (UniqueName: \"kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg\") pod \"5b250057-e693-476f-b75f-afb439b384a8\" (UID: \"5b250057-e693-476f-b75f-afb439b384a8\") " Nov 03 20:57:22.735297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.735269 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mds2\" (UniqueName: \"kubernetes.io/projected/da90ae73-1008-4edd-9b3e-c4011c77a46f-kube-api-access-5mds2\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:22.735479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.734226 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume" pod "5b250057-e693-476f-b75f-afb439b384a8" (UID: "5b250057-e693-476f-b75f-afb439b384a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:22.737367 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.737339 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg" pod "5b250057-e693-476f-b75f-afb439b384a8" (UID: "5b250057-e693-476f-b75f-afb439b384a8"). InnerVolumeSpecName "kube-api-access-lk8tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:22.837164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.837086 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk8tg\" (UniqueName: \"kubernetes.io/projected/5b250057-e693-476f-b75f-afb439b384a8-kube-api-access-lk8tg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:22.837164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:22.837115 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b250057-e693-476f-b75f-afb439b384a8-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:23.059491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.059454 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl"] Nov 03 20:57:23.064608 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.064578 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0acd8581-r6vtl"] Nov 03 20:57:23.081274 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.081242 5932 generic.go:356] "Generic (PLEG): container finished" podID="da90ae73-1008-4edd-9b3e-c4011c77a46f" containerID="77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb" exitCode=0 Nov 03 20:57:23.081735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.081301 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" Nov 03 20:57:23.081735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.081324 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" event={"ID":"da90ae73-1008-4edd-9b3e-c4011c77a46f","Type":"ContainerDied","Data":"77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb"} Nov 03 20:57:23.081735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.081374 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57" event={"ID":"da90ae73-1008-4edd-9b3e-c4011c77a46f","Type":"ContainerDied","Data":"d7b69a0e5777da762e06d6f8a2263792c03bf0ce55418c1b59d09e099b46c9c6"} Nov 03 20:57:23.081735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.081405 5932 scope.go:122] "RemoveContainer" containerID="77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb" Nov 03 20:57:23.083530 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.083506 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" event={"ID":"03d0cb91-0699-494a-9595-7d06a9c49e60","Type":"ContainerStarted","Data":"86dcd017b8e1a7601353d63cae759d39d27064a7f8f643653073c259b0150647"} Nov 03 20:57:23.083694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.083558 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" event={"ID":"03d0cb91-0699-494a-9595-7d06a9c49e60","Type":"ContainerStarted","Data":"f730679be0b1542705f0edb00698656d2f622de1daa8fee3b78cbddc154645a9"} Nov 03 20:57:23.085752 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.085728 5932 generic.go:356] "Generic (PLEG): container finished" podID="5b250057-e693-476f-b75f-afb439b384a8" containerID="5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1" exitCode=0 Nov 03 20:57:23.085874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.085781 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" Nov 03 20:57:23.085874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.085792 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" event={"ID":"5b250057-e693-476f-b75f-afb439b384a8","Type":"ContainerDied","Data":"5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1"} Nov 03 20:57:23.085874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.085825 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c" event={"ID":"5b250057-e693-476f-b75f-afb439b384a8","Type":"ContainerDied","Data":"ccb18b931c3c1f545ecaf1818e9e7b25486c99fd4a296b17275a5dd0209cbc6d"} Nov 03 20:57:23.096130 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.096110 5932 scope.go:122] "RemoveContainer" containerID="77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb" Nov 03 20:57:23.096418 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:57:23.096401 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb\": container with ID starting with 77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb not found: ID does not exist" containerID="77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb" Nov 03 20:57:23.096483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.096427 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb"} err="failed to get container status \"77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb\": rpc error: code = NotFound desc = could not find container \"77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb\": container with ID starting with 77e9deb64a0afa9ba2bedfcb2eeeafe29965f2215fda73246baa4d7ea407d6bb not found: ID does not exist" Nov 03 20:57:23.096483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.096443 5932 scope.go:122] "RemoveContainer" containerID="5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1" Nov 03 20:57:23.111120 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.111098 5932 scope.go:122] "RemoveContainer" containerID="5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1" Nov 03 20:57:23.111434 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:57:23.111413 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1\": container with ID starting with 5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1 not found: ID does not exist" containerID="5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1" Nov 03 20:57:23.111509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.111448 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1"} err="failed to get container status \"5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1\": rpc error: code = NotFound desc = could not find container \"5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1\": container with ID starting with 5494da6a80675878a48c1c8aa3202b625042c706804478a42668e660d30f89b1 not found: ID does not exist" Nov 03 20:57:23.247125 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.247059 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podStartSLOduration=2.247040686 podStartE2EDuration="2.247040686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:23.246434749 +0000 UTC m=+1651.311676224" watchObservedRunningTime="2026-11-03 20:57:23.247040686 +0000 UTC m=+1651.312282159" Nov 03 20:57:23.442942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.439715 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57"] Nov 03 20:57:23.451694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.449966 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d9a2c67d-775487bcbc-8gq57"] Nov 03 20:57:23.508649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.497681 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c"] Nov 03 20:57:23.518331 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.515484 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d9a2c67d-5f97b54f7f-sq49c"] Nov 03 20:57:23.687270 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.687192 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:23.692773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:23.692747 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:24.055799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.055720 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffdd307-0c75-4868-8f5f-f725b9b39c00" path="/var/lib/kubelet/pods/3ffdd307-0c75-4868-8f5f-f725b9b39c00/volumes" Nov 03 20:57:24.056145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.056130 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b250057-e693-476f-b75f-afb439b384a8" path="/var/lib/kubelet/pods/5b250057-e693-476f-b75f-afb439b384a8/volumes" Nov 03 20:57:24.056557 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.056544 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9176006-ca95-475c-b441-04e91b3a8861" path="/var/lib/kubelet/pods/b9176006-ca95-475c-b441-04e91b3a8861/volumes" Nov 03 20:57:24.057355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.057336 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c00eeb35-be75-470f-82ce-85c54a7b117d" path="/var/lib/kubelet/pods/c00eeb35-be75-470f-82ce-85c54a7b117d/volumes" Nov 03 20:57:24.057780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.057766 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da90ae73-1008-4edd-9b3e-c4011c77a46f" path="/var/lib/kubelet/pods/da90ae73-1008-4edd-9b3e-c4011c77a46f/volumes" Nov 03 20:57:24.099772 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.099747 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:24.100575 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:24.100557 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:57:28.062681 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:28.062608 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6"] Nov 03 20:57:28.097652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:28.097585 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2a5df515-mgmq6"] Nov 03 20:57:29.893649 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.892971 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79"] Nov 03 20:57:29.931067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.931028 5932 cpu_manager.go:414] "containerMap: removing container" podUID="da90ae73-1008-4edd-9b3e-c4011c77a46f" containerName="payload-processing" Nov 03 20:57:29.931333 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.931311 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da90ae73-1008-4edd-9b3e-c4011c77a46f" containerName="payload-processing" Nov 03 20:57:29.931501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.931489 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5b250057-e693-476f-b75f-afb439b384a8" containerName="payload-pre-processing" Nov 03 20:57:29.931581 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.931568 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b250057-e693-476f-b75f-afb439b384a8" containerName="payload-pre-processing" Nov 03 20:57:29.931987 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.931964 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="da90ae73-1008-4edd-9b3e-c4011c77a46f" containerName="payload-processing" Nov 03 20:57:29.932151 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.932129 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b250057-e693-476f-b75f-afb439b384a8" containerName="payload-pre-processing" Nov 03 20:57:29.977322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.970866 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79"] Nov 03 20:57:29.977322 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.970913 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7"] Nov 03 20:57:29.978215 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:29.978172 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.054707 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.001676 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.086647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.081304 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vggft\" (UniqueName: \"kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.086647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.081493 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.086647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.081558 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.086647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.081720 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6lg\" (UniqueName: \"kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.086647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.083466 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e68f8112\"" Nov 03 20:57:30.117727 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.099806 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e68f8112-dockercfg-ntj9r\"" Nov 03 20:57:30.227468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.227350 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6lg\" (UniqueName: \"kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.227873 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.227847 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vggft\" (UniqueName: \"kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.229099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.229062 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.229381 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.229350 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.230646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.230599 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.237306 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.237282 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.350679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.350026 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vggft\" (UniqueName: \"kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft\") pod \"payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.364364 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.364325 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc5dce0-f0f4-482f-bea9-97a728c74b00" path="/var/lib/kubelet/pods/5dc5dce0-f0f4-482f-bea9-97a728c74b00/volumes" Nov 03 20:57:30.369709 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.369600 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7"] Nov 03 20:57:30.370073 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.370055 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92"] Nov 03 20:57:30.377367 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.375769 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6lg\" (UniqueName: \"kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg\") pod \"payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.419470 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.419434 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1200aae-9kg92"] Nov 03 20:57:30.445809 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.445775 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:30.449723 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.449695 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:30.920938 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.919018 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79"] Nov 03 20:57:30.926754 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:30.926672 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7"] Nov 03 20:57:31.049499 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.049474 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj"] Nov 03 20:57:31.054451 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.054425 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b501cf3-q49bj"] Nov 03 20:57:31.411341 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.411296 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" event={"ID":"1f949d01-9ca7-4055-a159-91bb637bc3d3","Type":"ContainerStarted","Data":"71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec"} Nov 03 20:57:31.411529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.411370 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" event={"ID":"1f949d01-9ca7-4055-a159-91bb637bc3d3","Type":"ContainerStarted","Data":"d235fe585dd5c436d47e2b3efb93eb7a69514270fdb818f622004258b02e34bd"} Nov 03 20:57:31.411529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.411398 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:31.413431 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.413401 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" event={"ID":"41e72204-f86e-4db8-a1cf-5677653359ef","Type":"ContainerStarted","Data":"9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73"} Nov 03 20:57:31.413547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.413454 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" event={"ID":"41e72204-f86e-4db8-a1cf-5677653359ef","Type":"ContainerStarted","Data":"0e188a8c64240035d83116e041b3c4edd8b7acf1bb95069b539201e458d00da2"} Nov 03 20:57:31.413547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.413490 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:31.524203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:31.524152 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" podStartSLOduration=2.524138932 podStartE2EDuration="2.524138932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:31.519083402 +0000 UTC m=+1659.584324877" watchObservedRunningTime="2026-11-03 20:57:31.524138932 +0000 UTC m=+1659.589380404" Nov 03 20:57:32.054696 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:32.054664 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfbca07b-8e25-4d0f-bbe3-025dc66edf16" path="/var/lib/kubelet/pods/dfbca07b-8e25-4d0f-bbe3-025dc66edf16/volumes" Nov 03 20:57:32.055225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:32.055209 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0956623-54ef-4da1-9bad-0c7f1e07ff79" path="/var/lib/kubelet/pods/f0956623-54ef-4da1-9bad-0c7f1e07ff79/volumes" Nov 03 20:57:33.045773 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:33.045719 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" podStartSLOduration=4.045704224 podStartE2EDuration="4.045704224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:31.66830485 +0000 UTC m=+1659.733546323" watchObservedRunningTime="2026-11-03 20:57:33.045704224 +0000 UTC m=+1661.110945751" Nov 03 20:57:33.046923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:33.046897 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5"] Nov 03 20:57:33.093596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:33.093560 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e334666-d5xp5"] Nov 03 20:57:34.061219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:34.061186 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d633a502-cde9-4c81-8fd1-5d26c1b7b120" path="/var/lib/kubelet/pods/d633a502-cde9-4c81-8fd1-5d26c1b7b120/volumes" Nov 03 20:57:34.061580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:34.061565 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4"] Nov 03 20:57:34.100645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:34.098718 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ec1550-gtsh4"] Nov 03 20:57:36.114643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:36.113063 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9e75b1e-3047-4156-9827-c05450b4476f" path="/var/lib/kubelet/pods/e9e75b1e-3047-4156-9827-c05450b4476f/volumes" Nov 03 20:57:39.052440 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:39.052403 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf"] Nov 03 20:57:39.094097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:39.094070 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe1fdc9d-tkjvf"] Nov 03 20:57:40.058048 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:40.058011 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="380fdb76-75fc-4ea5-a3a8-2e995140667a" path="/var/lib/kubelet/pods/380fdb76-75fc-4ea5-a3a8-2e995140667a/volumes" Nov 03 20:57:41.050760 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:41.050724 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8"] Nov 03 20:57:41.093516 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:41.093481 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98dcbc6a-98fc8"] Nov 03 20:57:42.054513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:42.054472 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9246355-4a0a-4a2d-9616-e032bde5dffd" path="/var/lib/kubelet/pods/a9246355-4a0a-4a2d-9616-e032bde5dffd/volumes" Nov 03 20:57:42.426869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:42.426838 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 20:57:42.427267 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:42.426928 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 20:57:44.058673 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:44.058609 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh"] Nov 03 20:57:44.095143 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:44.095102 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-885e9d89-gkbvh"] Nov 03 20:57:44.153702 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:57:44.153665 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:57:44.893436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:44.893401 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb"] Nov 03 20:57:44.894485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:44.894421 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" containerID="cri-o://d275b1b10ba9158a1d007e3f7ed8226bb77fef3c1b2a1a7a4d0a3075a34ce285" gracePeriod=30 Nov 03 20:57:44.978038 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:44.977995 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd"] Nov 03 20:57:44.982915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:44.982889 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.038249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.036994 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:45.038249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.037046 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:57:45.038249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.038040 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd"] Nov 03 20:57:45.068483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.068429 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.068483 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.068482 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxnq\" (UniqueName: \"kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.069100 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.068589 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.170768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.170684 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.170768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.170755 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxnq\" (UniqueName: \"kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.170991 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.170822 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.171463 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.171436 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.173257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.173238 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.228194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.228157 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxnq\" (UniqueName: \"kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq\") pod \"maas-api-revoke-keys-e2e-derive-299a741b-jxzwd\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.318404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.318370 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:45.629887 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.629850 5932 generic.go:356] "Generic (PLEG): container finished" podID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerID="428e073bdb16da9a03daaec67437cf83307585811c5f94ff317be973f180c62d" exitCode=0 Nov 03 20:57:45.630086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.629932 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" event={"ID":"556c9d90-35c0-4dc1-a40c-8a0bf31b479b","Type":"ContainerDied","Data":"428e073bdb16da9a03daaec67437cf83307585811c5f94ff317be973f180c62d"} Nov 03 20:57:45.630488 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.630472 5932 scope.go:122] "RemoveContainer" containerID="428e073bdb16da9a03daaec67437cf83307585811c5f94ff317be973f180c62d" Nov 03 20:57:45.632273 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.632239 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:45.632386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.632323 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:45.632386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.632373 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:45.705181 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:45.705153 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd"] Nov 03 20:57:45.707253 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:57:45.707227 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844606a4_c13d_441b_877c_5699102cf549.slice/crio-e197d948096b0e8fbd9826d74e644103277313f2732098fd1b1f3f62d88d71f2 WatchSource:0}: Error finding container e197d948096b0e8fbd9826d74e644103277313f2732098fd1b1f3f62d88d71f2: Status 404 returned error can't find the container with id e197d948096b0e8fbd9826d74e644103277313f2732098fd1b1f3f62d88d71f2 Nov 03 20:57:46.055544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:46.055510 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdba8e76-bfc3-42d8-b4ee-cc722124f5a4" path="/var/lib/kubelet/pods/fdba8e76-bfc3-42d8-b4ee-cc722124f5a4/volumes" Nov 03 20:57:46.673074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:46.671566 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" event={"ID":"844606a4-c13d-441b-877c-5699102cf549","Type":"ContainerStarted","Data":"ace4cbb5431e5524119e071c6ee15633a77b65a6234aa245070476dd153cf2e9"} Nov 03 20:57:46.673074 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:46.671696 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" event={"ID":"844606a4-c13d-441b-877c-5699102cf549","Type":"ContainerStarted","Data":"e197d948096b0e8fbd9826d74e644103277313f2732098fd1b1f3f62d88d71f2"} Nov 03 20:57:46.679653 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:46.679172 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" event={"ID":"556c9d90-35c0-4dc1-a40c-8a0bf31b479b","Type":"ContainerStarted","Data":"b218f4aa2f6b4bc76de393d6c80030a83943c866c84c161c6746ddc53f4e3ad5"} Nov 03 20:57:46.788025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:46.787966 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" podStartSLOduration=2.787952778 podStartE2EDuration="2.787952778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:46.783405456 +0000 UTC m=+1674.848646933" watchObservedRunningTime="2026-11-03 20:57:46.787952778 +0000 UTC m=+1674.853194251" Nov 03 20:57:47.282919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.282830 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq"] Nov 03 20:57:47.286835 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.286808 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.347647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.347305 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-5b12d6-openshift-default-dockercfg-ht84s\"" Nov 03 20:57:47.360438 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.349170 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:57:47.361082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.360744 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:57:47.361082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.360804 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.469729 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.469790 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.469854 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.469879 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.469935 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.470006 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5slhx\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.470084 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.470188 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.470664 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.470253 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.496961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.496904 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq"] Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.671850 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5slhx\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.671972 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672072 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672138 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672380 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672422 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672489 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672513 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.672572 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.673132 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.674249 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.683456 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.691993 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.699320 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:47.996695 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.991387 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5slhx\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:48.012645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:47.999013 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:48.055704 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.051123 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:48.083321 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.083284 5932 generic.go:356] "Generic (PLEG): container finished" podID="844606a4-c13d-441b-877c-5699102cf549" containerID="ace4cbb5431e5524119e071c6ee15633a77b65a6234aa245070476dd153cf2e9" exitCode=0 Nov 03 20:57:48.089615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.088730 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token\") pod \"e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:48.106385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.106340 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" event={"ID":"844606a4-c13d-441b-877c-5699102cf549","Type":"ContainerDied","Data":"ace4cbb5431e5524119e071c6ee15633a77b65a6234aa245070476dd153cf2e9"} Nov 03 20:57:48.113391 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.113364 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:48.351085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.351004 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:57:48.351314 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.351285 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:57:48.603719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.603043 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq"] Nov 03 20:57:48.607145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.606952 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:48.607145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.607020 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:48.607145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:48.607053 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:57:49.096637 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.096594 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" event={"ID":"9b2828a1-58e3-4cfa-8ac7-760aab663beb","Type":"ContainerStarted","Data":"ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035"} Nov 03 20:57:49.097100 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.096690 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" event={"ID":"9b2828a1-58e3-4cfa-8ac7-760aab663beb","Type":"ContainerStarted","Data":"26b6ba946248469c9ac7f206bcc828383ebafed1e140225557262f9fb956274f"} Nov 03 20:57:49.114265 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.114234 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:49.115454 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.115429 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:57:49.115598 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.115484 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:57:49.220004 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.219951 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podStartSLOduration=2.219934544 podStartE2EDuration="2.219934544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:49.215353296 +0000 UTC m=+1677.280594770" watchObservedRunningTime="2026-11-03 20:57:49.219934544 +0000 UTC m=+1677.285176018" Nov 03 20:57:49.370522 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.370019 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:57:49.370522 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.370083 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:57:49.388650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.385847 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:49.577747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.567426 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca\") pod \"844606a4-c13d-441b-877c-5699102cf549\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " Nov 03 20:57:49.577747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.567509 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca\") pod \"844606a4-c13d-441b-877c-5699102cf549\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " Nov 03 20:57:49.577747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.567540 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq\" (UniqueName: \"kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq\") pod \"844606a4-c13d-441b-877c-5699102cf549\" (UID: \"844606a4-c13d-441b-877c-5699102cf549\") " Nov 03 20:57:49.579344 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.579285 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca" pod "844606a4-c13d-441b-877c-5699102cf549" (UID: "844606a4-c13d-441b-877c-5699102cf549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:49.582996 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.582925 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca" pod "844606a4-c13d-441b-877c-5699102cf549" (UID: "844606a4-c13d-441b-877c-5699102cf549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:49.627022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.619310 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq" pod "844606a4-c13d-441b-877c-5699102cf549" (UID: "844606a4-c13d-441b-877c-5699102cf549"). InnerVolumeSpecName "kube-api-access-scxnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:49.678662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.671436 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/844606a4-c13d-441b-877c-5699102cf549-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.678662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.671476 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/844606a4-c13d-441b-877c-5699102cf549-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:49.678662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:49.671493 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxnq\" (UniqueName: \"kubernetes.io/projected/844606a4-c13d-441b-877c-5699102cf549-kube-api-access-scxnq\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.108796 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.108762 5932 generic.go:356] "Generic (PLEG): container finished" podID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerID="d275b1b10ba9158a1d007e3f7ed8226bb77fef3c1b2a1a7a4d0a3075a34ce285" exitCode=0 Nov 03 20:57:50.109278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.108887 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" event={"ID":"b8c273ee-2cca-417b-a2be-4132c3a271be","Type":"ContainerDied","Data":"d275b1b10ba9158a1d007e3f7ed8226bb77fef3c1b2a1a7a4d0a3075a34ce285"} Nov 03 20:57:50.110881 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.110856 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" Nov 03 20:57:50.111032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.110906 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd" event={"ID":"844606a4-c13d-441b-877c-5699102cf549","Type":"ContainerDied","Data":"e197d948096b0e8fbd9826d74e644103277313f2732098fd1b1f3f62d88d71f2"} Nov 03 20:57:50.111032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.110951 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e197d948096b0e8fbd9826d74e644103277313f2732098fd1b1f3f62d88d71f2" Nov 03 20:57:50.118360 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.118340 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:50.192226 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.192167 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:57:50.259702 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.259671 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.259866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.259739 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.259866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.259768 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.259866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.259807 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.260015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.259943 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.260015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.259991 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:50.260134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260075 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.260134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260109 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.260315 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260193 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.260315 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260099 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:50.260448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260346 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:50.260448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260399 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket\") pod \"b8c273ee-2cca-417b-a2be-4132c3a271be\" (UID: \"b8c273ee-2cca-417b-a2be-4132c3a271be\") " Nov 03 20:57:50.260780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260751 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:50.260886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.260790 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:50.261368 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.261340 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.261461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.261372 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.261461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.261387 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.261461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.261400 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.261461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.261414 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c273ee-2cca-417b-a2be-4132c3a271be-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.262237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.262216 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:57:50.262396 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.262381 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:57:50.262595 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.262562 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "kube-api-access-mgdsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:50.262718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.262604 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token" pod "b8c273ee-2cca-417b-a2be-4132c3a271be" (UID: "b8c273ee-2cca-417b-a2be-4132c3a271be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:50.351671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.349301 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:57:50.351671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.349360 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.354685 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk"] Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.356012 5932 cpu_manager.go:414] "containerMap: removing container" podUID="844606a4-c13d-441b-877c-5699102cf549" containerName="revoke-keys" Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.356033 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844606a4-c13d-441b-877c-5699102cf549" containerName="revoke-keys" Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.356127 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.356137 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.356333 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="844606a4-c13d-441b-877c-5699102cf549" containerName="revoke-keys" Nov 03 20:57:50.360901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.356365 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" containerName="istio-proxy" Nov 03 20:57:50.362993 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.362971 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk"] Nov 03 20:57:50.363145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.363080 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.363510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.363477 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.363510 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.363514 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdsm\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-kube-api-access-mgdsm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.363771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.363531 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.363771 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.363545 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c273ee-2cca-417b-a2be-4132c3a271be-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:50.450694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.449056 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:57:50.467646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.462565 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 20:57:50.474046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.470408 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lvvm\" (UniqueName: \"kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.474046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.470812 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.474046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.470878 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.572090 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.572048 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.572316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.572123 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.572316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.572201 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lvvm\" (UniqueName: \"kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.572955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.572928 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.574806 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.574783 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.645660 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.641082 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:57:50.647087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.647060 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lvvm\" (UniqueName: \"kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm\") pod \"maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:50.692642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:50.688910 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:51.142378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.124174 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" Nov 03 20:57:51.142378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.129843 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb" event={"ID":"b8c273ee-2cca-417b-a2be-4132c3a271be","Type":"ContainerDied","Data":"02e80b8c3c9b5a2782c5e34db7bfd27025516ecf23476d56cd745daab21e36c9"} Nov 03 20:57:51.142378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.129909 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:51.142378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.129945 5932 scope.go:122] "RemoveContainer" containerID="d275b1b10ba9158a1d007e3f7ed8226bb77fef3c1b2a1a7a4d0a3075a34ce285" Nov 03 20:57:51.142378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.131342 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:57:51.264710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.264687 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk"] Nov 03 20:57:51.267042 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:57:51.267008 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d369aff_1318_4bfe_9c1b_d7a3b4dbbb04.slice/crio-3fefee222e8af9194af534b255c85fd8416895f0a799cf9692ac923ac9b51cd5 WatchSource:0}: Error finding container 3fefee222e8af9194af534b255c85fd8416895f0a799cf9692ac923ac9b51cd5: Status 404 returned error can't find the container with id 3fefee222e8af9194af534b255c85fd8416895f0a799cf9692ac923ac9b51cd5 Nov 03 20:57:51.303689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.303661 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb"] Nov 03 20:57:51.310418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.310396 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-vr6kb"] Nov 03 20:57:51.349021 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.348979 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:57:51.349210 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:51.349039 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:57:52.059099 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:52.059023 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c273ee-2cca-417b-a2be-4132c3a271be" path="/var/lib/kubelet/pods/b8c273ee-2cca-417b-a2be-4132c3a271be/volumes" Nov 03 20:57:52.157979 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:52.157945 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" event={"ID":"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04","Type":"ContainerStarted","Data":"881c6c350dcdd80c3ab860b3908c0781888838d02557879d7fe3bd4e38aed860"} Nov 03 20:57:52.158396 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:52.158000 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" event={"ID":"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04","Type":"ContainerStarted","Data":"3fefee222e8af9194af534b255c85fd8416895f0a799cf9692ac923ac9b51cd5"} Nov 03 20:57:52.273676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:52.271387 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" podStartSLOduration=2.271373295 podStartE2EDuration="2.271373295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:57:52.268262781 +0000 UTC m=+1680.333504252" watchObservedRunningTime="2026-11-03 20:57:52.271373295 +0000 UTC m=+1680.336614777" Nov 03 20:57:52.348906 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:52.348815 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 03 20:57:52.348906 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:52.348882 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 03 20:57:53.170678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:53.170596 5932 generic.go:356] "Generic (PLEG): container finished" podID="9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" containerID="881c6c350dcdd80c3ab860b3908c0781888838d02557879d7fe3bd4e38aed860" exitCode=0 Nov 03 20:57:53.171087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:53.170677 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" event={"ID":"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04","Type":"ContainerDied","Data":"881c6c350dcdd80c3ab860b3908c0781888838d02557879d7fe3bd4e38aed860"} Nov 03 20:57:53.354719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:53.354688 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:57:53.356117 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:53.356094 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:57:54.369339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.369311 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:54.484513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.484435 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca\") pod \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " Nov 03 20:57:54.484703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.484593 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca\") pod \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " Nov 03 20:57:54.484703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.484689 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm\" (UniqueName: \"kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm\") pod \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\" (UID: \"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04\") " Nov 03 20:57:54.484920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.484885 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca" pod "9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" (UID: "9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:57:54.485944 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.485916 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:54.486961 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.486936 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca" pod "9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" (UID: "9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:57:54.490050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.490011 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm" pod "9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" (UID: "9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04"). InnerVolumeSpecName "kube-api-access-8lvvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:57:54.587541 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.587500 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:54.587541 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:54.587537 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lvvm\" (UniqueName: \"kubernetes.io/projected/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04-kube-api-access-8lvvm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:57:55.049131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.049089 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8"] Nov 03 20:57:55.095216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.095183 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0151c44b-rdlj8"] Nov 03 20:57:55.119781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.119742 5932 scope.go:122] "RemoveContainer" containerID="dff5de1b62f8e2e4c52a583a0d5869eaeb5fed0045f01728b5d57b351fb6b3c1" Nov 03 20:57:55.132403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.132379 5932 scope.go:122] "RemoveContainer" containerID="7d2e56d7a4d7433cb6644e4d14bb69acfde65a0fd79709033f7e8a1af2b81328" Nov 03 20:57:55.143711 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.143677 5932 scope.go:122] "RemoveContainer" containerID="f2404d0e940bb6a974544be53c53392c8e3bdf217ab3c165779952a3abe2c390" Nov 03 20:57:55.154915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.154888 5932 scope.go:122] "RemoveContainer" containerID="c83119d103db057da8df7615e093b8f7738803a14336d5e893cb7f1c8b6a08cc" Nov 03 20:57:55.166688 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.166665 5932 scope.go:122] "RemoveContainer" containerID="7d8fca06b7f2c9d8bcbd167bc42253f657f818646897caf11300e52955cbf217" Nov 03 20:57:55.177723 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.177697 5932 scope.go:122] "RemoveContainer" containerID="89661fb10a66cd6d0b1b6ae825857fbf7189feb941525ef3bf7a72c3eff6cdd1" Nov 03 20:57:55.195842 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.195814 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" Nov 03 20:57:55.195973 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.195820 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk" event={"ID":"9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04","Type":"ContainerDied","Data":"3fefee222e8af9194af534b255c85fd8416895f0a799cf9692ac923ac9b51cd5"} Nov 03 20:57:55.196029 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:55.196000 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fefee222e8af9194af534b255c85fd8416895f0a799cf9692ac923ac9b51cd5" Nov 03 20:57:56.056339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:56.056305 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f34b31-3fb8-4f26-83b2-a955443251eb" path="/var/lib/kubelet/pods/83f34b31-3fb8-4f26-83b2-a955443251eb/volumes" Nov 03 20:57:57.048000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:57.047969 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb"] Nov 03 20:57:57.093693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:57.093655 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adeeb386-s2dqb"] Nov 03 20:57:57.175285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:57.175252 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k"] Nov 03 20:57:57.298967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:57.295669 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv"] Nov 03 20:57:58.054731 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:58.054697 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97a7b8d3-b8db-4aed-853e-4e05ab491cf8" path="/var/lib/kubelet/pods/97a7b8d3-b8db-4aed-853e-4e05ab491cf8/volumes" Nov 03 20:57:59.077657 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:59.072607 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn"] Nov 03 20:57:59.090587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:57:59.088925 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4953da8e-rrsbn"] Nov 03 20:58:00.054804 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:00.054769 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2091b247-7369-41a1-9559-3ce1defa7538" path="/var/lib/kubelet/pods/2091b247-7369-41a1-9559-3ce1defa7538/volumes" Nov 03 20:58:00.055225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:00.055202 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9"] Nov 03 20:58:00.093498 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:00.093458 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a3cb6a7-7hzd9"] Nov 03 20:58:01.557443 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.557405 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc"] Nov 03 20:58:01.559161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.559135 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" containerName="revoke-keys" Nov 03 20:58:01.560131 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.560109 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" containerName="revoke-keys" Nov 03 20:58:01.560643 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.560598 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" containerName="revoke-keys" Nov 03 20:58:01.578647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.575650 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:01.581976 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.581937 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7jp\" (UniqueName: \"kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp\") pod \"payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:01.582433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.582407 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:01.641412 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.640067 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-5b12d6\"" Nov 03 20:58:01.675645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.658300 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-5b12d6-dockercfg-ptjlx\"" Nov 03 20:58:01.676997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.676960 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc"] Nov 03 20:58:01.713025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.699549 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:01.713025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.699773 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7jp\" (UniqueName: \"kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp\") pod \"payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:01.718651 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.717973 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:01.967646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:01.964417 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7jp\" (UniqueName: \"kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp\") pod \"payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:02.011204 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.011171 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:02.127405 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.127372 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51249b85-a099-4e74-a455-f539b60af19a" path="/var/lib/kubelet/pods/51249b85-a099-4e74-a455-f539b60af19a/volumes" Nov 03 20:58:02.176556 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.176471 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" podUID="77275cb9-5cde-42eb-ba3b-b842aa392759" containerName="payload-processing" containerID="cri-o://a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33" gracePeriod=25 Nov 03 20:58:02.291497 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.291425 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f"] Nov 03 20:58:02.296025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.295976 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" podUID="160e8882-42ee-46ee-8df9-336323a0b1f0" containerName="payload-pre-processing" containerID="cri-o://6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615" gracePeriod=25 Nov 03 20:58:02.392862 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.392779 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq"] Nov 03 20:58:02.469149 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.469122 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:58:02.517545 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.517514 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl\" (UniqueName: \"kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl\") pod \"77275cb9-5cde-42eb-ba3b-b842aa392759\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " Nov 03 20:58:02.517733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.517560 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume\") pod \"77275cb9-5cde-42eb-ba3b-b842aa392759\" (UID: \"77275cb9-5cde-42eb-ba3b-b842aa392759\") " Nov 03 20:58:02.518703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.518294 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume" pod "77275cb9-5cde-42eb-ba3b-b842aa392759" (UID: "77275cb9-5cde-42eb-ba3b-b842aa392759"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:02.518703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.518591 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77275cb9-5cde-42eb-ba3b-b842aa392759-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:02.521313 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.521287 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl" pod "77275cb9-5cde-42eb-ba3b-b842aa392759" (UID: "77275cb9-5cde-42eb-ba3b-b842aa392759"). InnerVolumeSpecName "kube-api-access-4fvrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:02.524725 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:58:02.524699 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8a3ea4_6468_482e_87c3_c82190f5717b.slice/crio-cfa7012cfa61e94062352602fd56ed314a0e7c13b4a7938fbb5d9e3e9cdd2d65 WatchSource:0}: Error finding container cfa7012cfa61e94062352602fd56ed314a0e7c13b4a7938fbb5d9e3e9cdd2d65: Status 404 returned error can't find the container with id cfa7012cfa61e94062352602fd56ed314a0e7c13b4a7938fbb5d9e3e9cdd2d65 Nov 03 20:58:02.525532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.525505 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc"] Nov 03 20:58:02.596015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.595986 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:58:02.619475 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.619449 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume\") pod \"160e8882-42ee-46ee-8df9-336323a0b1f0\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " Nov 03 20:58:02.620186 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.619734 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv\" (UniqueName: \"kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv\") pod \"160e8882-42ee-46ee-8df9-336323a0b1f0\" (UID: \"160e8882-42ee-46ee-8df9-336323a0b1f0\") " Nov 03 20:58:02.620186 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.619939 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume" pod "160e8882-42ee-46ee-8df9-336323a0b1f0" (UID: "160e8882-42ee-46ee-8df9-336323a0b1f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:02.620297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.620202 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/160e8882-42ee-46ee-8df9-336323a0b1f0-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:02.620297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.620222 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fvrl\" (UniqueName: \"kubernetes.io/projected/77275cb9-5cde-42eb-ba3b-b842aa392759-kube-api-access-4fvrl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:02.621746 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.621728 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv" pod "160e8882-42ee-46ee-8df9-336323a0b1f0" (UID: "160e8882-42ee-46ee-8df9-336323a0b1f0"). InnerVolumeSpecName "kube-api-access-8j8nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:02.735562 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:02.732449 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j8nv\" (UniqueName: \"kubernetes.io/projected/160e8882-42ee-46ee-8df9-336323a0b1f0-kube-api-access-8j8nv\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:03.362225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.362091 5932 generic.go:356] "Generic (PLEG): container finished" podID="160e8882-42ee-46ee-8df9-336323a0b1f0" containerID="6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615" exitCode=0 Nov 03 20:58:03.362467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.362243 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" event={"ID":"160e8882-42ee-46ee-8df9-336323a0b1f0","Type":"ContainerDied","Data":"6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615"} Nov 03 20:58:03.362467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.362304 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" event={"ID":"160e8882-42ee-46ee-8df9-336323a0b1f0","Type":"ContainerDied","Data":"8a301e3c2318ce207a8db49311be81c0c39d7bc965aa0e8d0938ef71ec06ea9f"} Nov 03 20:58:03.362467 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.362336 5932 scope.go:122] "RemoveContainer" containerID="6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615" Nov 03 20:58:03.362831 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.362703 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv" Nov 03 20:58:03.376867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.374509 5932 generic.go:356] "Generic (PLEG): container finished" podID="77275cb9-5cde-42eb-ba3b-b842aa392759" containerID="a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33" exitCode=0 Nov 03 20:58:03.376867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.374921 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" event={"ID":"77275cb9-5cde-42eb-ba3b-b842aa392759","Type":"ContainerDied","Data":"a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33"} Nov 03 20:58:03.376867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.374994 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" event={"ID":"77275cb9-5cde-42eb-ba3b-b842aa392759","Type":"ContainerDied","Data":"c6b34a33e50c0a17956d05e86f6121f8a2171d4e2876c7567799910c3990002d"} Nov 03 20:58:03.376867 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.375076 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k" Nov 03 20:58:03.431900 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.429736 5932 scope.go:122] "RemoveContainer" containerID="6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615" Nov 03 20:58:03.437088 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:58:03.435707 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615\": container with ID starting with 6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615 not found: ID does not exist" containerID="6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615" Nov 03 20:58:03.437088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.435752 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615"} err="failed to get container status \"6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615\": rpc error: code = NotFound desc = could not find container \"6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615\": container with ID starting with 6e272217a0cdd03296de952ac406092b2fc513076e808f7ab80ea6e950ad0615 not found: ID does not exist" Nov 03 20:58:03.437088 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.435776 5932 scope.go:122] "RemoveContainer" containerID="a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33" Nov 03 20:58:03.441339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.441290 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" event={"ID":"9e8a3ea4-6468-482e-87c3-c82190f5717b","Type":"ContainerStarted","Data":"1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8"} Nov 03 20:58:03.441486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.441370 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" event={"ID":"9e8a3ea4-6468-482e-87c3-c82190f5717b","Type":"ContainerStarted","Data":"cfa7012cfa61e94062352602fd56ed314a0e7c13b4a7938fbb5d9e3e9cdd2d65"} Nov 03 20:58:03.441655 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.441492 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:03.456042 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.456019 5932 scope.go:122] "RemoveContainer" containerID="a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33" Nov 03 20:58:03.456362 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:58:03.456334 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33\": container with ID starting with a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33 not found: ID does not exist" containerID="a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33" Nov 03 20:58:03.456445 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.456372 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33"} err="failed to get container status \"a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33\": rpc error: code = NotFound desc = could not find container \"a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33\": container with ID starting with a48da17268b3240cdb3c677ce2fb0485c13c2f3321bc37480572cc2df1a3bd33 not found: ID does not exist" Nov 03 20:58:03.533899 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.533870 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k"] Nov 03 20:58:03.541697 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.541668 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-299a741b-95dd6c5bf-nn72k"] Nov 03 20:58:03.643618 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.643551 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv"] Nov 03 20:58:03.648346 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.648308 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-299a741b-55f7ddff8c-9fgfv"] Nov 03 20:58:03.765640 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:03.755969 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" podStartSLOduration=2.755951692 podStartE2EDuration="2.755951692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:03.754818216 +0000 UTC m=+1691.820059691" watchObservedRunningTime="2026-11-03 20:58:03.755951692 +0000 UTC m=+1691.821193166" Nov 03 20:58:04.146656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:04.139334 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="160e8882-42ee-46ee-8df9-336323a0b1f0" path="/var/lib/kubelet/pods/160e8882-42ee-46ee-8df9-336323a0b1f0/volumes" Nov 03 20:58:04.146656 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:04.140196 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77275cb9-5cde-42eb-ba3b-b842aa392759" path="/var/lib/kubelet/pods/77275cb9-5cde-42eb-ba3b-b842aa392759/volumes" Nov 03 20:58:07.291796 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.291753 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" podUID="1ca2e14f-72f0-428c-b56e-d10850407dfc" containerName="payload-processing" containerID="cri-o://ae1880dfadb911cc8e118c9ec2244ef8feb660be50f1c4da8547273c26aeb898" gracePeriod=25 Nov 03 20:58:07.393810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.393772 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" podUID="937486ff-920a-492b-8e55-e9cf2cbf14e5" containerName="payload-pre-processing" containerID="cri-o://b780f695e5009bfd440b049d1acb6c0b6d1f7883005e3b18ef444c87c5b716d5" gracePeriod=25 Nov 03 20:58:07.666777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.666544 5932 generic.go:356] "Generic (PLEG): container finished" podID="937486ff-920a-492b-8e55-e9cf2cbf14e5" containerID="b780f695e5009bfd440b049d1acb6c0b6d1f7883005e3b18ef444c87c5b716d5" exitCode=0 Nov 03 20:58:07.666777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.666700 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" event={"ID":"937486ff-920a-492b-8e55-e9cf2cbf14e5","Type":"ContainerDied","Data":"b780f695e5009bfd440b049d1acb6c0b6d1f7883005e3b18ef444c87c5b716d5"} Nov 03 20:58:07.680679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.679893 5932 generic.go:356] "Generic (PLEG): container finished" podID="1ca2e14f-72f0-428c-b56e-d10850407dfc" containerID="ae1880dfadb911cc8e118c9ec2244ef8feb660be50f1c4da8547273c26aeb898" exitCode=0 Nov 03 20:58:07.680679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.679958 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" event={"ID":"1ca2e14f-72f0-428c-b56e-d10850407dfc","Type":"ContainerDied","Data":"ae1880dfadb911cc8e118c9ec2244ef8feb660be50f1c4da8547273c26aeb898"} Nov 03 20:58:07.757129 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.752817 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:58:07.793615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.793583 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll"] Nov 03 20:58:07.817468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.817443 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:58:07.830491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.830457 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d9a2c67d-9bpll"] Nov 03 20:58:07.862148 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.862070 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb\" (UniqueName: \"kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb\") pod \"937486ff-920a-492b-8e55-e9cf2cbf14e5\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " Nov 03 20:58:07.862280 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.862260 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume\") pod \"1ca2e14f-72f0-428c-b56e-d10850407dfc\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " Nov 03 20:58:07.862325 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.862291 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume\") pod \"937486ff-920a-492b-8e55-e9cf2cbf14e5\" (UID: \"937486ff-920a-492b-8e55-e9cf2cbf14e5\") " Nov 03 20:58:07.862378 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.862358 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz\" (UniqueName: \"kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz\") pod \"1ca2e14f-72f0-428c-b56e-d10850407dfc\" (UID: \"1ca2e14f-72f0-428c-b56e-d10850407dfc\") " Nov 03 20:58:07.862764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.862738 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume" pod "937486ff-920a-492b-8e55-e9cf2cbf14e5" (UID: "937486ff-920a-492b-8e55-e9cf2cbf14e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:07.862764 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.862747 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume" pod "1ca2e14f-72f0-428c-b56e-d10850407dfc" (UID: "1ca2e14f-72f0-428c-b56e-d10850407dfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:07.863033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.863017 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ca2e14f-72f0-428c-b56e-d10850407dfc-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:07.863033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.863034 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/937486ff-920a-492b-8e55-e9cf2cbf14e5-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:07.864272 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.864248 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb" pod "937486ff-920a-492b-8e55-e9cf2cbf14e5" (UID: "937486ff-920a-492b-8e55-e9cf2cbf14e5"). InnerVolumeSpecName "kube-api-access-njqcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:07.864389 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.864368 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz" pod "1ca2e14f-72f0-428c-b56e-d10850407dfc" (UID: "1ca2e14f-72f0-428c-b56e-d10850407dfc"). InnerVolumeSpecName "kube-api-access-l2jhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:07.964670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.964599 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jhz\" (UniqueName: \"kubernetes.io/projected/1ca2e14f-72f0-428c-b56e-d10850407dfc-kube-api-access-l2jhz\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:07.964670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:07.964669 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njqcb\" (UniqueName: \"kubernetes.io/projected/937486ff-920a-492b-8e55-e9cf2cbf14e5-kube-api-access-njqcb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:08.058386 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.058353 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68913a47-fec4-4dce-9ab2-eb94f35ac138" path="/var/lib/kubelet/pods/68913a47-fec4-4dce-9ab2-eb94f35ac138/volumes" Nov 03 20:58:08.691234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.691208 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" Nov 03 20:58:08.691234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.691222 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq" event={"ID":"937486ff-920a-492b-8e55-e9cf2cbf14e5","Type":"ContainerDied","Data":"8eab87670175b68e17cebce6bb12d1e6b68500a98a4b5b576f34901fb8c68917"} Nov 03 20:58:08.691805 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.691288 5932 scope.go:122] "RemoveContainer" containerID="b780f695e5009bfd440b049d1acb6c0b6d1f7883005e3b18ef444c87c5b716d5" Nov 03 20:58:08.692910 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.692884 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" event={"ID":"1ca2e14f-72f0-428c-b56e-d10850407dfc","Type":"ContainerDied","Data":"9abb55afaab4b74dc055eda23064db8baee3461407268f0da79b9e342772f38b"} Nov 03 20:58:08.693030 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.692898 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f" Nov 03 20:58:08.703080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.703064 5932 scope.go:122] "RemoveContainer" containerID="ae1880dfadb911cc8e118c9ec2244ef8feb660be50f1c4da8547273c26aeb898" Nov 03 20:58:08.848243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.848207 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq"] Nov 03 20:58:08.894086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.894054 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df4313ad-774b996d47-h2dbq"] Nov 03 20:58:08.984194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.984109 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f"] Nov 03 20:58:08.989320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:08.989295 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df4313ad-6f97984c9f-l6d6f"] Nov 03 20:58:09.049079 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:09.049044 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx"] Nov 03 20:58:09.053447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:09.053423 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d2cb38-j5mmx"] Nov 03 20:58:10.054670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:10.054605 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca2e14f-72f0-428c-b56e-d10850407dfc" path="/var/lib/kubelet/pods/1ca2e14f-72f0-428c-b56e-d10850407dfc/volumes" Nov 03 20:58:10.055385 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:10.055362 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937486ff-920a-492b-8e55-e9cf2cbf14e5" path="/var/lib/kubelet/pods/937486ff-920a-492b-8e55-e9cf2cbf14e5/volumes" Nov 03 20:58:10.055965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:10.055944 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15be16f-9973-48bb-ae38-618c04442d61" path="/var/lib/kubelet/pods/d15be16f-9973-48bb-ae38-618c04442d61/volumes" Nov 03 20:58:10.056390 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:10.056371 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b"] Nov 03 20:58:10.056490 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:10.056402 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e8162b-lsw6b"] Nov 03 20:58:12.199645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:12.194386 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7147335-1917-4975-b8a1-f66f39b0bf7f" path="/var/lib/kubelet/pods/c7147335-1917-4975-b8a1-f66f39b0bf7f/volumes" Nov 03 20:58:14.184616 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:58:14.184573 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:58:14.518503 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:14.518433 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 20:58:19.232776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:19.232738 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:58:19.233251 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:19.232792 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:58:22.035228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.035131 5932 generic.go:356] "Generic (PLEG): container finished" podID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerID="66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387" exitCode=0 Nov 03 20:58:22.035228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.035186 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" event={"ID":"2314c3a7-3dbe-4574-9c32-42175696ec7c","Type":"ContainerDied","Data":"66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387"} Nov 03 20:58:22.035725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.035701 5932 scope.go:122] "RemoveContainer" containerID="66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387" Nov 03 20:58:22.037683 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.037651 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:22.037839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.037758 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:22.037839 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.037810 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.891841 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw"] Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893254 5932 cpu_manager.go:414] "containerMap: removing container" podUID="937486ff-920a-492b-8e55-e9cf2cbf14e5" containerName="payload-pre-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893280 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937486ff-920a-492b-8e55-e9cf2cbf14e5" containerName="payload-pre-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893325 5932 cpu_manager.go:414] "containerMap: removing container" podUID="160e8882-42ee-46ee-8df9-336323a0b1f0" containerName="payload-pre-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893334 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="160e8882-42ee-46ee-8df9-336323a0b1f0" containerName="payload-pre-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893362 5932 cpu_manager.go:414] "containerMap: removing container" podUID="77275cb9-5cde-42eb-ba3b-b842aa392759" containerName="payload-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893370 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77275cb9-5cde-42eb-ba3b-b842aa392759" containerName="payload-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893397 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1ca2e14f-72f0-428c-b56e-d10850407dfc" containerName="payload-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893404 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca2e14f-72f0-428c-b56e-d10850407dfc" containerName="payload-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893602 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="937486ff-920a-492b-8e55-e9cf2cbf14e5" containerName="payload-pre-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893638 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="160e8882-42ee-46ee-8df9-336323a0b1f0" containerName="payload-pre-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893651 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="77275cb9-5cde-42eb-ba3b-b842aa392759" containerName="payload-processing" Nov 03 20:58:22.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.893664 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca2e14f-72f0-428c-b56e-d10850407dfc" containerName="payload-processing" Nov 03 20:58:22.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:22.940113 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.120694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.078183 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.120694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.078254 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.120694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.078325 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.120694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.078360 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gsl\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.120694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.078438 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.120694 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.078604 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.123237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.121588 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.123237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.121705 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.123237 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.121812 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.187751 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.186898 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ba7e83-openshift-default-dockercfg-xzmfl\"" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.231917 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232012 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232079 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232138 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232191 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gsl\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232306 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232509 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232541 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232654 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.232896 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.239687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.233149 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.241421 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.241384 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.309692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.242812 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.309692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.243295 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.309692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.289137 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.309692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.296414 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.326612 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.326570 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" event={"ID":"2314c3a7-3dbe-4574-9c32-42175696ec7c","Type":"ContainerStarted","Data":"165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c"} Nov 03 20:58:23.495911 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.488235 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:58:23.578602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.576646 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:58:23.578602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.576714 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:58:23.678085 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.678053 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gsl\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.689466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.685163 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token\") pod \"e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:23.719705 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.718190 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw"] Nov 03 20:58:23.834227 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:23.825607 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:24.362917 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:24.362883 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw"] Nov 03 20:58:24.367663 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:58:24.367635 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9e6dfb_9c04_4894_808f_d8364790e47b.slice/crio-370d88ba98e678c5777d8586a1a07cc667731f2f8951c8dbb1cf3429f8bd2581 WatchSource:0}: Error finding container 370d88ba98e678c5777d8586a1a07cc667731f2f8951c8dbb1cf3429f8bd2581: Status 404 returned error can't find the container with id 370d88ba98e678c5777d8586a1a07cc667731f2f8951c8dbb1cf3429f8bd2581 Nov 03 20:58:24.376535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:24.376483 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:24.376717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:24.376567 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:24.376717 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:24.376598 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:24.452223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:24.452183 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:58:24.452479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:24.452250 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:58:25.403569 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.403528 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" event={"ID":"af9e6dfb-9c04-4894-808f-d8364790e47b","Type":"ContainerStarted","Data":"9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b"} Nov 03 20:58:25.404092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.403598 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" event={"ID":"af9e6dfb-9c04-4894-808f-d8364790e47b","Type":"ContainerStarted","Data":"370d88ba98e678c5777d8586a1a07cc667731f2f8951c8dbb1cf3429f8bd2581"} Nov 03 20:58:25.451926 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.451888 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:58:25.452144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.451950 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:58:25.604509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.604458 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podStartSLOduration=3.604443682 podStartE2EDuration="3.604443682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:25.562100659 +0000 UTC m=+1713.627342132" watchObservedRunningTime="2026-11-03 20:58:25.604443682 +0000 UTC m=+1713.669685154" Nov 03 20:58:25.836735 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.833228 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:25.845646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:25.843004 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:26.465644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:26.464773 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:58:26.465644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:26.464836 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:58:26.471970 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:26.468269 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:26.472858 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:26.472836 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 20:58:27.451838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:27.451797 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:58:27.451838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:27.451863 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:58:28.452272 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:28.452236 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:58:28.452725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:28.452299 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Nov 03 20:58:29.456886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:29.456852 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:58:29.457363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:29.457328 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:58:29.458060 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:29.458039 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 20:58:35.489769 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:35.489673 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk"] Nov 03 20:58:35.535308 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:35.535276 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-df4313ad-c5tbk"] Nov 03 20:58:36.056885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:36.056850 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04" path="/var/lib/kubelet/pods/9d369aff-1318-4bfe-9c1b-d7a3b4dbbb04/volumes" Nov 03 20:58:36.606563 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:36.606498 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd"] Nov 03 20:58:36.618534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:36.618500 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-299a741b-jxzwd"] Nov 03 20:58:36.636291 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:36.634200 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh"] Nov 03 20:58:36.643479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:36.637141 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" containerID="cri-o://21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d" gracePeriod=30 Nov 03 20:58:37.154297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:37.154230 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh"] Nov 03 20:58:37.155570 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:37.155519 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" containerID="cri-o://b218f4aa2f6b4bc76de393d6c80030a83943c866c84c161c6746ddc53f4e3ad5" gracePeriod=30 Nov 03 20:58:38.406611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.406088 5932 patch_prober.go:29] interesting pod/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:58:38.406611 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.406171 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:58:38.429967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.429923 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844606a4-c13d-441b-877c-5699102cf549" path="/var/lib/kubelet/pods/844606a4-c13d-441b-877c-5699102cf549/volumes" Nov 03 20:58:38.514836 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.514799 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq"] Nov 03 20:58:38.518593 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.518567 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.576667 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.575516 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-e427e7-openshift-default-dockercfg-qrxbf\"" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.633908 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xhbn\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.633990 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634029 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634093 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634225 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634275 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634304 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634477 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.641647 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.634564 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737501 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737606 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737712 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xhbn\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737767 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737803 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737857 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.737982 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.738032 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.738056 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.738960 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.746699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.739335 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.777774 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.777226 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.779283 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.779255 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.781813 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.780165 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.781813 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.780858 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq"] Nov 03 20:58:38.878035 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.877982 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:38.905997 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:38.905913 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:39.015809 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.003705 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:39.063648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.063397 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xhbn\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn\") pod \"e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:39.211777 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.211742 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:39.671080 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.671053 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq"] Nov 03 20:58:39.674226 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:58:39.674190 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcdb585_bf18_4dd5_a2dd_45a14c591520.slice/crio-aa71f062e615f1083ceef919a5a4092e18dfaf687b37b3180619ee4d5440ede6 WatchSource:0}: Error finding container aa71f062e615f1083ceef919a5a4092e18dfaf687b37b3180619ee4d5440ede6: Status 404 returned error can't find the container with id aa71f062e615f1083ceef919a5a4092e18dfaf687b37b3180619ee4d5440ede6 Nov 03 20:58:39.677209 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.677171 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:39.677309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.677248 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:39.677309 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:39.677282 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:58:40.373049 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:40.373015 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" event={"ID":"5fcdb585-bf18-4dd5-a2dd-45a14c591520","Type":"ContainerStarted","Data":"57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09"} Nov 03 20:58:40.373254 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:40.373081 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" event={"ID":"5fcdb585-bf18-4dd5-a2dd-45a14c591520","Type":"ContainerStarted","Data":"aa71f062e615f1083ceef919a5a4092e18dfaf687b37b3180619ee4d5440ede6"} Nov 03 20:58:40.491860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:40.491808 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podStartSLOduration=2.4917934170000002 podStartE2EDuration="2.491793417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:40.489396842 +0000 UTC m=+1728.554638317" watchObservedRunningTime="2026-11-03 20:58:40.491793417 +0000 UTC m=+1728.557034949" Nov 03 20:58:41.212799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:41.212768 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:41.218133 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:41.218106 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:41.383260 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:41.383230 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:41.384207 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:41.384183 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 20:58:42.037679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.037599 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:58:42.159532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.159497 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.159532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.159549 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.159920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.159605 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.159920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.159724 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.159920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.159901 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.160057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.159978 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.160057 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.160033 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.160132 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.160092 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.160188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.160142 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo\") pod \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\" (UID: \"94be4b6a-4a95-40e7-9ead-b4f02fd232c5\") " Nov 03 20:58:42.161300 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.161249 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.161442 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.161319 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.161745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.161727 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.161882 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.161676 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.162493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.162440 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:42.164255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.164222 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:42.184403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.184364 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:42.184542 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.184511 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "kube-api-access-86dnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:42.184724 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.184562 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy" pod "94be4b6a-4a95-40e7-9ead-b4f02fd232c5" (UID: "94be4b6a-4a95-40e7-9ead-b4f02fd232c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.262950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262877 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.262950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262905 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.262950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262920 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.262950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262933 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.262950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262942 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.262950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262951 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.263449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262961 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.263449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262970 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86dnz\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-kube-api-access-86dnz\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.263449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.262979 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94be4b6a-4a95-40e7-9ead-b4f02fd232c5-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.401582 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.401534 5932 generic.go:356] "Generic (PLEG): container finished" podID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerID="b218f4aa2f6b4bc76de393d6c80030a83943c866c84c161c6746ddc53f4e3ad5" exitCode=0 Nov 03 20:58:42.401799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.401647 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" event={"ID":"556c9d90-35c0-4dc1-a40c-8a0bf31b479b","Type":"ContainerDied","Data":"b218f4aa2f6b4bc76de393d6c80030a83943c866c84c161c6746ddc53f4e3ad5"} Nov 03 20:58:42.401799 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.401705 5932 scope.go:122] "RemoveContainer" containerID="428e073bdb16da9a03daaec67437cf83307585811c5f94ff317be973f180c62d" Nov 03 20:58:42.404481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.404456 5932 generic.go:356] "Generic (PLEG): container finished" podID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerID="21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d" exitCode=0 Nov 03 20:58:42.404698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.404506 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" event={"ID":"94be4b6a-4a95-40e7-9ead-b4f02fd232c5","Type":"ContainerDied","Data":"21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d"} Nov 03 20:58:42.404698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.404533 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" Nov 03 20:58:42.404698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.404582 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh" event={"ID":"94be4b6a-4a95-40e7-9ead-b4f02fd232c5","Type":"ContainerDied","Data":"67fccc66dd4d4698e585f345d75dceee1d571d8aab451f162dc734fe048b88e0"} Nov 03 20:58:42.426250 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.425916 5932 scope.go:122] "RemoveContainer" containerID="21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d" Nov 03 20:58:42.440420 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.440400 5932 scope.go:122] "RemoveContainer" containerID="9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb" Nov 03 20:58:42.443755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.443735 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:58:42.452774 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.452753 5932 scope.go:122] "RemoveContainer" containerID="21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d" Nov 03 20:58:42.453134 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:58:42.453114 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d\": container with ID starting with 21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d not found: ID does not exist" containerID="21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d" Nov 03 20:58:42.453234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.453144 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d"} err="failed to get container status \"21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d\": rpc error: code = NotFound desc = could not find container \"21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d\": container with ID starting with 21731af06a4170e98b3502c0f6fa7ef104b6df83ecf7a3ee521ed380e8ea515d not found: ID does not exist" Nov 03 20:58:42.453234 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.453167 5932 scope.go:122] "RemoveContainer" containerID="9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb" Nov 03 20:58:42.453503 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:58:42.453480 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb\": container with ID starting with 9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb not found: ID does not exist" containerID="9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb" Nov 03 20:58:42.453567 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.453510 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb"} err="failed to get container status \"9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb\": rpc error: code = NotFound desc = could not find container \"9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb\": container with ID starting with 9f024cfeeb94a8ea5e74e64604b5250aaa025e2ad1ce788fd3e0cba3b3a884bb not found: ID does not exist" Nov 03 20:58:42.568689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568566 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.568689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568602 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.568689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568685 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.568930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568739 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.568930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568827 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.568930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568855 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.568930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568878 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.568930 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.568901 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.569195 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569018 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.569258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569021 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.569258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569174 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:58:42.569258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569231 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data\") pod \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\" (UID: \"556c9d90-35c0-4dc1-a40c-8a0bf31b479b\") " Nov 03 20:58:42.569432 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569186 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.569486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569445 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.569919 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569899 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.570032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569920 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.570032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569933 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.570032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569946 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.570032 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.569959 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.571525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.571498 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:58:42.572097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.572068 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:58:42.572248 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.572229 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:42.572307 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.572264 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn" pod "556c9d90-35c0-4dc1-a40c-8a0bf31b479b" (UID: "556c9d90-35c0-4dc1-a40c-8a0bf31b479b"). InnerVolumeSpecName "kube-api-access-4sdzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:58:42.671228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.671193 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.671228 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.671226 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdzn\" (UniqueName: \"kubernetes.io/projected/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-kube-api-access-4sdzn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.671447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.671239 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.671447 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.671249 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/556c9d90-35c0-4dc1-a40c-8a0bf31b479b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:58:42.679932 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.679901 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh"] Nov 03 20:58:42.684319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:42.684294 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-299a741b-openshift-default-6fb74d7cb9-mqsnh"] Nov 03 20:58:43.056319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.056288 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w"] Nov 03 20:58:43.066311 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.066285 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f727b7f-wpl4w"] Nov 03 20:58:43.416973 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.416932 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" event={"ID":"556c9d90-35c0-4dc1-a40c-8a0bf31b479b","Type":"ContainerDied","Data":"59d8c57a76bd14571f5579f1b2681520719f80434a3e7ae21354d1633b06609b"} Nov 03 20:58:43.417418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.417008 5932 scope.go:122] "RemoveContainer" containerID="b218f4aa2f6b4bc76de393d6c80030a83943c866c84c161c6746ddc53f4e3ad5" Nov 03 20:58:43.417418 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.417179 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh" Nov 03 20:58:43.579413 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.579375 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh"] Nov 03 20:58:43.626294 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:43.626262 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-df4313ad-openshift-default-8c4c578b5-qq9sh"] Nov 03 20:58:44.055558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:44.055525 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b195c12-c233-4638-983b-383700b4c6bb" path="/var/lib/kubelet/pods/0b195c12-c233-4638-983b-383700b4c6bb/volumes" Nov 03 20:58:44.056161 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:44.056144 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" path="/var/lib/kubelet/pods/556c9d90-35c0-4dc1-a40c-8a0bf31b479b/volumes" Nov 03 20:58:44.056886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:44.056859 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" path="/var/lib/kubelet/pods/94be4b6a-4a95-40e7-9ead-b4f02fd232c5/volumes" Nov 03 20:58:44.154595 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:58:44.154548 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:58:46.988801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:46.988412 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26"] Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003499 5932 cpu_manager.go:414] "containerMap: removing container" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003527 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003573 5932 cpu_manager.go:414] "containerMap: removing container" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003581 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003635 5932 cpu_manager.go:414] "containerMap: removing container" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003650 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.003996 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.004015 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.004033 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="94be4b6a-4a95-40e7-9ead-b4f02fd232c5" containerName="istio-proxy" Nov 03 20:58:47.008644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.004044 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" Nov 03 20:58:47.050645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.048587 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26"] Nov 03 20:58:47.050645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.049072 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.095645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.091204 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.095645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.091255 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lz54\" (UniqueName: \"kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54\") pod \"payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.200646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.195008 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e427e7\"" Nov 03 20:58:47.200646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.195348 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e427e7-dockercfg-lxmpq\"" Nov 03 20:58:47.247972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.211213 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.247972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.211268 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lz54\" (UniqueName: \"kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54\") pod \"payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.247972 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.213168 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.304015 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.303981 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs"] Nov 03 20:58:47.365650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.357981 5932 cpu_manager.go:414] "containerMap: removing container" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" Nov 03 20:58:47.365888 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.365862 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="556c9d90-35c0-4dc1-a40c-8a0bf31b479b" containerName="istio-proxy" Nov 03 20:58:47.386150 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.386105 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.388548 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.388519 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs"] Nov 03 20:58:47.448602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.448480 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf872\" (UniqueName: \"kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872\") pod \"payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.448602 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.448557 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.508698 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.502177 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lz54\" (UniqueName: \"kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54\") pod \"payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.550725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.550665 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf872\" (UniqueName: \"kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872\") pod \"payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.551026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.551002 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.552375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.552011 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.610205 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.610172 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf872\" (UniqueName: \"kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872\") pod \"payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:47.610439 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.610425 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:47.843886 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:47.843807 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:48.016355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.016323 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26"] Nov 03 20:58:48.020708 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:58:48.020681 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9f7bfc_acbc_4554_9e5b_e8a2aab509c5.slice/crio-4ea62e754dc0c3d8355425aac6cc2b97bf3d933cd562a0141a1d287238700c43 WatchSource:0}: Error finding container 4ea62e754dc0c3d8355425aac6cc2b97bf3d933cd562a0141a1d287238700c43: Status 404 returned error can't find the container with id 4ea62e754dc0c3d8355425aac6cc2b97bf3d933cd562a0141a1d287238700c43 Nov 03 20:58:48.225453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.225426 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs"] Nov 03 20:58:48.228574 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:58:48.228547 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fba486a_efd8_47e1_893a_d408c3daeee9.slice/crio-c27945c627f2b0578b97ae1b758018f0d71a29ef132bce35f4d9f8b3ecfb5a85 WatchSource:0}: Error finding container c27945c627f2b0578b97ae1b758018f0d71a29ef132bce35f4d9f8b3ecfb5a85: Status 404 returned error can't find the container with id c27945c627f2b0578b97ae1b758018f0d71a29ef132bce35f4d9f8b3ecfb5a85 Nov 03 20:58:48.561699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.561659 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" event={"ID":"2fba486a-efd8-47e1-893a-d408c3daeee9","Type":"ContainerStarted","Data":"983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d"} Nov 03 20:58:48.561892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.561722 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" event={"ID":"2fba486a-efd8-47e1-893a-d408c3daeee9","Type":"ContainerStarted","Data":"c27945c627f2b0578b97ae1b758018f0d71a29ef132bce35f4d9f8b3ecfb5a85"} Nov 03 20:58:48.561892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.561762 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:48.563486 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.563460 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" event={"ID":"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5","Type":"ContainerStarted","Data":"e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0"} Nov 03 20:58:48.563587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.563503 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" event={"ID":"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5","Type":"ContainerStarted","Data":"4ea62e754dc0c3d8355425aac6cc2b97bf3d933cd562a0141a1d287238700c43"} Nov 03 20:58:48.563587 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.563523 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:58:48.670784 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.670734 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" podStartSLOduration=1.670720548 podStartE2EDuration="1.670720548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:48.669286624 +0000 UTC m=+1736.734528098" watchObservedRunningTime="2026-11-03 20:58:48.670720548 +0000 UTC m=+1736.735962012" Nov 03 20:58:48.780056 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:48.780006 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" podStartSLOduration=2.779991924 podStartE2EDuration="2.779991924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:58:48.778664186 +0000 UTC m=+1736.843905661" watchObservedRunningTime="2026-11-03 20:58:48.779991924 +0000 UTC m=+1736.845233396" Nov 03 20:58:55.282951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.282914 5932 scope.go:122] "RemoveContainer" containerID="b4944c06b3ca4b1e4b0221c7cf4d7e32309c07a8552546f1f19d1a4f64ebbc66" Nov 03 20:58:55.295398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.295375 5932 scope.go:122] "RemoveContainer" containerID="fe2cc2d8f16886f92f31744596ad0c501029b96a44c3a756c16c3de3196fd928" Nov 03 20:58:55.305403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.305385 5932 scope.go:122] "RemoveContainer" containerID="c7a32f086d3f29d73ded5733a32055e7064fe0bdb61593f53e13da051093e823" Nov 03 20:58:55.315513 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.315487 5932 scope.go:122] "RemoveContainer" containerID="aa186e472e517ecba7748dab9a6e6887f726aecf2d4e114a66555220738fe446" Nov 03 20:58:55.325354 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.325336 5932 scope.go:122] "RemoveContainer" containerID="91cbad3418d8ec155b7ff63c047a5ea6ca6cb509cda893371ca472406fd35303" Nov 03 20:58:55.334967 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.334945 5932 scope.go:122] "RemoveContainer" containerID="f435236fad741a0952e4ea9091e4058908a616ae33161ab3188362c1e1764673" Nov 03 20:58:55.346423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.346406 5932 scope.go:122] "RemoveContainer" containerID="cb475fe150b974f53be294d51f194f4dbbc4d1a039cf40b8d83d40dd713e59ed" Nov 03 20:58:55.356291 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.356272 5932 scope.go:122] "RemoveContainer" containerID="f7dc477aa2c1857e4b303f7bf46c2dac2627c5fb8e2e02d5863b5e9ac8d0850d" Nov 03 20:58:55.365679 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.365658 5932 scope.go:122] "RemoveContainer" containerID="2427681c6d40b80b612c7a951c7001f1e9db0840fac1c047e3fabcd74f893bc3" Nov 03 20:58:55.375144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.375116 5932 scope.go:122] "RemoveContainer" containerID="733da3ca8cbab3a5c199ee60f4e317d8f7ee6ce715b4a64a6ab8b2317c5d91da" Nov 03 20:58:55.384786 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.384766 5932 scope.go:122] "RemoveContainer" containerID="0e5df1b9d715fd15690ea7d98cc4ff2292bc4950b8fdd07aa5321a352a063d93" Nov 03 20:58:55.397863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.397845 5932 scope.go:122] "RemoveContainer" containerID="19a1c443b44cbf2dc1818c35fee1947e9d740e48362c63d8e658510235e9f92f" Nov 03 20:58:55.408495 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.408476 5932 scope.go:122] "RemoveContainer" containerID="d862c1d79f7e741b0a5e9b41f19ff8ab8c21f8694e156ac906a5d9252f563f2a" Nov 03 20:58:55.418874 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.418853 5932 scope.go:122] "RemoveContainer" containerID="7b30892aef707dfd43994f1fa1ee39f7eb2c57b561b76f14afa0b9df676f80c8" Nov 03 20:58:55.428461 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.428442 5932 scope.go:122] "RemoveContainer" containerID="746da95743988880b77fb9644a22d3f6e4478df6420a4d82d777abcf94e06a2e" Nov 03 20:58:55.437459 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.437439 5932 scope.go:122] "RemoveContainer" containerID="8e1ecfad45f88d2cadbfcf059bc2c300d3a7e859f6aeb16aff2e78d4687d3168" Nov 03 20:58:55.446969 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.446950 5932 scope.go:122] "RemoveContainer" containerID="5ce3dc355689e7c72bf155c51f2f986d663835c00e8daccb0541da1162344d91" Nov 03 20:58:55.457861 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:55.457843 5932 scope.go:122] "RemoveContainer" containerID="547f8f347f1c7f927b861f75e197273115b633319ab2a172ad7057415b0c1027" Nov 03 20:58:57.944070 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:57.944036 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p"] Nov 03 20:58:57.944701 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:57.944295 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" containerID="cri-o://6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c" gracePeriod=30 Nov 03 20:58:57.989964 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:57.989936 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf"] Nov 03 20:58:57.990170 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:57.990149 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" containerID="cri-o://eda28d3386368d36b72faad2b330a0cc8f53a78044ab06b2c99c77d4c9c75f41" gracePeriod=30 Nov 03 20:58:57.994460 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:57.994434 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq"] Nov 03 20:58:57.994665 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:57.994641 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" containerID="cri-o://261f2cf849c0454a2b201608f46114caadf5b4e43858b6f0f8222140b4cfe0d9" gracePeriod=30 Nov 03 20:58:58.036255 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:58.036217 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c"] Nov 03 20:58:58.036481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:58.036457 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" containerID="cri-o://6f0d9797c5a41f15e43f102ee40da2a23aaf674899d3f8323d486983d6381812" gracePeriod=30 Nov 03 20:58:59.577800 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:59.577770 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 20:58:59.578181 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:58:59.577857 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 20:59:01.355719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:01.353814 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:59:01.355719 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:01.353894 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:59:02.603737 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:02.603696 5932 patch_prober.go:29] interesting pod/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:59:02.604108 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:02.603754 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:59:02.803444 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:02.803406 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d"] Nov 03 20:59:02.812221 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:02.812202 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:02.921020 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:02.920983 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-9ce672-openshift-default-dockercfg-jbrfj\"" Nov 03 20:59:03.180693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.180214 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.180693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.180298 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.180693 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.180401 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.308703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.308354 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.308703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.308539 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.308703 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.308685 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.308990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.308766 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvmfw\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.308990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.308814 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.308990 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.308838 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.314305 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.313096 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d"] Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410520 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410638 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvmfw\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410682 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410707 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410800 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410870 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.410984 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.411056 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.411178 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.411734 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.412921 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.412513 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.416320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.413711 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.416320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.413981 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.416320 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.414224 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.420879 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.420856 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.431374 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.431318 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.477112 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.476906 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvmfw\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.482989 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.482233 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token\") pod \"e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.653160 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.651904 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.725535 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.725606 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.725672 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.725718 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.725868 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.725955 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.726086 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.726140 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.726255 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.744310 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.729170 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:03.807525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.753496 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:03.807525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.755000 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.807525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.755125 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "kube-api-access-fvgq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.807525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.770865 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.807525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.771098 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.807525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.788737 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.817423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.808272 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.817423 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.814977 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:03.838219 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.838184 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.851188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.851154 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\") pod \"f4d854bc-b23f-4902-9985-ded9777dcf0a\" (UID: \"f4d854bc-b23f-4902-9985-ded9777dcf0a\") " Nov 03 20:59:03.859362 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:59:03.859336 5932 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f4d854bc-b23f-4902-9985-ded9777dcf0a/volumes/kubernetes.io~empty-dir/istio-envoy Nov 03 20:59:03.859596 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.859551 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy" pod "f4d854bc-b23f-4902-9985-ded9777dcf0a" (UID: "f4d854bc-b23f-4902-9985-ded9777dcf0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.859782 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.859765 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4d854bc-b23f-4902-9985-ded9777dcf0a-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.859885 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.859876 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.859949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.859938 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvgq8\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-kube-api-access-fvgq8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860009 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860002 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860059 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860052 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860144 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860131 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860230 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860218 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860332 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860320 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860420 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860408 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4d854bc-b23f-4902-9985-ded9777dcf0a-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:03.860477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860408 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" event={"ID":"f0782bf0-b851-4167-ab65-38ba0be007d2","Type":"ContainerDied","Data":"eda28d3386368d36b72faad2b330a0cc8f53a78044ab06b2c99c77d4c9c75f41"} Nov 03 20:59:03.860566 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.860371 5932 generic.go:356] "Generic (PLEG): container finished" podID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerID="eda28d3386368d36b72faad2b330a0cc8f53a78044ab06b2c99c77d4c9c75f41" exitCode=0 Nov 03 20:59:03.866026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.865994 5932 generic.go:356] "Generic (PLEG): container finished" podID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerID="6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c" exitCode=0 Nov 03 20:59:03.866143 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.866121 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" event={"ID":"f4d854bc-b23f-4902-9985-ded9777dcf0a","Type":"ContainerDied","Data":"6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c"} Nov 03 20:59:03.866203 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.866172 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" event={"ID":"f4d854bc-b23f-4902-9985-ded9777dcf0a","Type":"ContainerDied","Data":"68e84b6b48a7eb4f1c256a21c8383d6544623d423e1295710e41590f25769c5a"} Nov 03 20:59:03.866257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.866201 5932 scope.go:122] "RemoveContainer" containerID="6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c" Nov 03 20:59:03.866395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.866378 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p" Nov 03 20:59:03.871025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.870711 5932 generic.go:356] "Generic (PLEG): container finished" podID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerID="261f2cf849c0454a2b201608f46114caadf5b4e43858b6f0f8222140b4cfe0d9" exitCode=0 Nov 03 20:59:03.871025 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.870838 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" event={"ID":"4d3e97ff-af2c-4099-94b1-e5d45b325b2d","Type":"ContainerDied","Data":"261f2cf849c0454a2b201608f46114caadf5b4e43858b6f0f8222140b4cfe0d9"} Nov 03 20:59:03.872597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.872569 5932 generic.go:356] "Generic (PLEG): container finished" podID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerID="6f0d9797c5a41f15e43f102ee40da2a23aaf674899d3f8323d486983d6381812" exitCode=0 Nov 03 20:59:03.872875 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.872814 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" event={"ID":"dd9ee5ad-f460-40e9-9a88-721b89da9f86","Type":"ContainerDied","Data":"6f0d9797c5a41f15e43f102ee40da2a23aaf674899d3f8323d486983d6381812"} Nov 03 20:59:03.875222 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.874463 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:59:03.878188 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.878171 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:59:03.894666 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.885014 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:59:03.903953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.903929 5932 scope.go:122] "RemoveContainer" containerID="6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c" Nov 03 20:59:03.904375 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:03.904350 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c\": container with ID starting with 6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c not found: ID does not exist" containerID="6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c" Nov 03 20:59:03.904493 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.904385 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c"} err="failed to get container status \"6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c\": rpc error: code = NotFound desc = could not find container \"6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c\": container with ID starting with 6c341227b77fa0b0850579ea90021b1881ab79c0909e1a4a979e0858f511673c not found: ID does not exist" Nov 03 20:59:03.961206 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961176 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.961377 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961226 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.961377 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961260 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.961377 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961329 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.961377 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961352 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961389 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961433 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961454 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961484 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961507 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961552 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961578 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.961614 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961604 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961659 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961682 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961703 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961731 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961808 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961892 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961912 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961940 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961973 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.961995 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data\") pod \"f0782bf0-b851-4167-ab65-38ba0be007d2\" (UID: \"f0782bf0-b851-4167-ab65-38ba0be007d2\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.962012 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.962042 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert\") pod \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\" (UID: \"dd9ee5ad-f460-40e9-9a88-721b89da9f86\") " Nov 03 20:59:03.962067 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.962064 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.962815 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.962094 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy\") pod \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\" (UID: \"4d3e97ff-af2c-4099-94b1-e5d45b325b2d\") " Nov 03 20:59:03.963212 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.963184 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.963801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.963416 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.963801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.963446 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.963801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.963747 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.964037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.963866 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:03.964037 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.963886 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:03.964718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.964695 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.964818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.964609 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.964953 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.964927 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.965066 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.965040 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.965789 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.965765 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.965983 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.965963 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.966054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.965991 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.966054 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.966017 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.966245 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.966221 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:03.966528 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.966502 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.968395 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.968359 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.968615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.968587 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.968951 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.968925 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "kube-api-access-vbmxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.969116 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.969095 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:03.969272 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.969253 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:03.969466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.969449 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "kube-api-access-hdxxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.969534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.969522 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "kube-api-access-qd6zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.970257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.970232 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:03.970257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.970238 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo" pod "f0782bf0-b851-4167-ab65-38ba0be007d2" (UID: "f0782bf0-b851-4167-ab65-38ba0be007d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:03.971026 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.971008 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token" pod "dd9ee5ad-f460-40e9-9a88-721b89da9f86" (UID: "dd9ee5ad-f460-40e9-9a88-721b89da9f86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:03.971096 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:03.971027 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token" pod "4d3e97ff-af2c-4099-94b1-e5d45b325b2d" (UID: "4d3e97ff-af2c-4099-94b1-e5d45b325b2d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:04.068369 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068334 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0782bf0-b851-4167-ab65-38ba0be007d2-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068370 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd6zp\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-kube-api-access-qd6zp\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068387 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068401 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068415 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068427 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068441 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068456 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068470 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068482 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068496 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068509 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.068520 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068523 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068536 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068566 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068581 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmxh\" (UniqueName: \"kubernetes.io/projected/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-kube-api-access-vbmxh\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068596 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068609 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068638 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068652 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxxb\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-kube-api-access-hdxxb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068667 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068681 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068694 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068708 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068721 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd9ee5ad-f460-40e9-9a88-721b89da9f86-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068735 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3e97ff-af2c-4099-94b1-e5d45b325b2d-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.069145 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.068749 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0782bf0-b851-4167-ab65-38ba0be007d2-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:04.261223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.261186 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p"] Nov 03 20:59:04.266472 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.266442 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-bjr5p"] Nov 03 20:59:04.310115 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.310043 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d"] Nov 03 20:59:04.314528 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:59:04.314497 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a834d62_437d_4854_a49d_a23856e72225.slice/crio-2aef70dad606054dd660f21de5587e70a7a8856dfafd97b3c8b75f91d7e1da87 WatchSource:0}: Error finding container 2aef70dad606054dd660f21de5587e70a7a8856dfafd97b3c8b75f91d7e1da87: Status 404 returned error can't find the container with id 2aef70dad606054dd660f21de5587e70a7a8856dfafd97b3c8b75f91d7e1da87 Nov 03 20:59:04.317419 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.317391 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:04.317505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.317449 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:04.317505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.317481 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:04.885086 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.885045 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" event={"ID":"5a834d62-437d-4854-a49d-a23856e72225","Type":"ContainerStarted","Data":"d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d"} Nov 03 20:59:04.885543 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.885115 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" event={"ID":"5a834d62-437d-4854-a49d-a23856e72225","Type":"ContainerStarted","Data":"2aef70dad606054dd660f21de5587e70a7a8856dfafd97b3c8b75f91d7e1da87"} Nov 03 20:59:04.886699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.886669 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" Nov 03 20:59:04.886813 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.886710 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf" event={"ID":"f0782bf0-b851-4167-ab65-38ba0be007d2","Type":"ContainerDied","Data":"0d0409efd4e70dcb26d6c60264aebb5dd60eca5b4803fa6ee606238d241a9991"} Nov 03 20:59:04.886813 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.886758 5932 scope.go:122] "RemoveContainer" containerID="eda28d3386368d36b72faad2b330a0cc8f53a78044ab06b2c99c77d4c9c75f41" Nov 03 20:59:04.889725 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.889701 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" event={"ID":"4d3e97ff-af2c-4099-94b1-e5d45b325b2d","Type":"ContainerDied","Data":"6899f504250df9b95676cc1fe632b543775c9ff26d9ac5d821e955d6e11f874e"} Nov 03 20:59:04.889828 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.889771 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq" Nov 03 20:59:04.891373 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.891354 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" Nov 03 20:59:04.891479 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.891355 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c" event={"ID":"dd9ee5ad-f460-40e9-9a88-721b89da9f86","Type":"ContainerDied","Data":"ea6bccf32cf3cce76e1d8113421e1a94e502cf00b2d9c9776a7b781ff8ee1bb6"} Nov 03 20:59:04.900063 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.900045 5932 scope.go:122] "RemoveContainer" containerID="261f2cf849c0454a2b201608f46114caadf5b4e43858b6f0f8222140b4cfe0d9" Nov 03 20:59:04.910687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:04.910668 5932 scope.go:122] "RemoveContainer" containerID="6f0d9797c5a41f15e43f102ee40da2a23aaf674899d3f8323d486983d6381812" Nov 03 20:59:05.000346 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.000286 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podStartSLOduration=3.000268466 podStartE2EDuration="3.000268466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:04.998396306 +0000 UTC m=+1753.063637780" watchObservedRunningTime="2026-11-03 20:59:05.000268466 +0000 UTC m=+1753.065509940" Nov 03 20:59:05.108107 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.108077 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c"] Nov 03 20:59:05.112339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.112311 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-hvp4c"] Nov 03 20:59:05.281515 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.281408 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq"] Nov 03 20:59:05.288949 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.288914 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-27mwq"] Nov 03 20:59:05.380303 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.380264 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf"] Nov 03 20:59:05.385484 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.385455 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-88228a-openshift-default-6c9d56f9c7-45bvf"] Nov 03 20:59:05.796642 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.787688 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:05.991253 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:05.991221 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:06.082915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:06.082829 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" path="/var/lib/kubelet/pods/4d3e97ff-af2c-4099-94b1-e5d45b325b2d/volumes" Nov 03 20:59:06.087848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:06.087793 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" path="/var/lib/kubelet/pods/dd9ee5ad-f460-40e9-9a88-721b89da9f86/volumes" Nov 03 20:59:06.088968 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:06.088939 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" path="/var/lib/kubelet/pods/f0782bf0-b851-4167-ab65-38ba0be007d2/volumes" Nov 03 20:59:06.089860 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:06.089841 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" path="/var/lib/kubelet/pods/f4d854bc-b23f-4902-9985-ded9777dcf0a/volumes" Nov 03 20:59:06.957407 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:06.957374 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:06.958257 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:06.958232 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 20:59:12.238194 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:12.238160 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz"] Nov 03 20:59:12.239645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:12.239606 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37b51cfa-qmtlz"] Nov 03 20:59:13.049558 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:13.049518 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp"] Nov 03 20:59:13.049923 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:13.049860 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" containerID="cri-o://d81dfab08b900af97b81a1bf0526262460075f2a7e2575451ac4f44979e2a1e3" gracePeriod=30 Nov 03 20:59:13.079676 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:13.079516 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx"] Nov 03 20:59:13.080165 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:13.080106 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" containerID="cri-o://f95d30db4e42440be84837d49ddf5fddb207ebdec337b1d1e2a043bca952698b" gracePeriod=30 Nov 03 20:59:13.085915 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:13.085888 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r"] Nov 03 20:59:13.086092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:13.086058 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" containerID="cri-o://974401bb5ad969553fc0e1b40a98f4aa6e0f22edca768219849baa9fe4784c49" gracePeriod=30 Nov 03 20:59:14.056196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:14.056151 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c27f29b-89ad-491c-83cf-7678d518fa95" path="/var/lib/kubelet/pods/8c27f29b-89ad-491c-83cf-7678d518fa95/volumes" Nov 03 20:59:14.154081 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:14.154040 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:59:18.305398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.304810 5932 generic.go:356] "Generic (PLEG): container finished" podID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerID="ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035" exitCode=0 Nov 03 20:59:18.305398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.304887 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" event={"ID":"9b2828a1-58e3-4cfa-8ac7-760aab663beb","Type":"ContainerDied","Data":"ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035"} Nov 03 20:59:18.306000 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.305420 5932 scope.go:122] "RemoveContainer" containerID="ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035" Nov 03 20:59:18.306927 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.306721 5932 generic.go:356] "Generic (PLEG): container finished" podID="ede56218-956c-417d-a362-23f3e37f0f8d" containerID="d81dfab08b900af97b81a1bf0526262460075f2a7e2575451ac4f44979e2a1e3" exitCode=0 Nov 03 20:59:18.306927 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.306851 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" event={"ID":"ede56218-956c-417d-a362-23f3e37f0f8d","Type":"ContainerDied","Data":"d81dfab08b900af97b81a1bf0526262460075f2a7e2575451ac4f44979e2a1e3"} Nov 03 20:59:18.307453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.307420 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:18.307538 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.307509 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:18.307582 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.307551 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:18.311375 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.311356 5932 generic.go:356] "Generic (PLEG): container finished" podID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerID="f95d30db4e42440be84837d49ddf5fddb207ebdec337b1d1e2a043bca952698b" exitCode=0 Nov 03 20:59:18.311554 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.311535 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" event={"ID":"c2d88b32-d6e3-4077-a805-2d88867c93ad","Type":"ContainerDied","Data":"f95d30db4e42440be84837d49ddf5fddb207ebdec337b1d1e2a043bca952698b"} Nov 03 20:59:18.313414 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.313392 5932 generic.go:356] "Generic (PLEG): container finished" podID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerID="974401bb5ad969553fc0e1b40a98f4aa6e0f22edca768219849baa9fe4784c49" exitCode=0 Nov 03 20:59:18.313532 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.313518 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" event={"ID":"4297f2be-be32-4a9a-bd71-c1d294e1f569","Type":"ContainerDied","Data":"974401bb5ad969553fc0e1b40a98f4aa6e0f22edca768219849baa9fe4784c49"} Nov 03 20:59:18.409509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.409473 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:59:18.435858 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.435782 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:59:18.527123 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527086 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.527278 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527190 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527277 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527355 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527338 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527367 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.527468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527436 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.527468 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527464 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.527689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527489 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527512 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527530 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.527689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527548 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527689 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527606 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.527955 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.527888 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.528339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.528295 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.528456 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.528421 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.528521 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.528482 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.528965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.528938 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.529371 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529352 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.529485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529464 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.529662 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529617 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.529747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529706 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.529807 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529755 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.529868 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529848 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.529922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529904 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.529982 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529938 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy\") pod \"ede56218-956c-417d-a362-23f3e37f0f8d\" (UID: \"ede56218-956c-417d-a362-23f3e37f0f8d\") " Nov 03 20:59:18.530036 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.529982 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy\") pod \"c2d88b32-d6e3-4077-a805-2d88867c93ad\" (UID: \"c2d88b32-d6e3-4077-a805-2d88867c93ad\") " Nov 03 20:59:18.531041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.530810 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.530834 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.530848 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531041 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.530864 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.531154 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:18.531297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.530881 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531297 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.531207 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.531222 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.531692 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.531499 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.535102 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.535065 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:18.537937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.537900 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.545913 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.545921 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "kube-api-access-kxmng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.545938 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "kube-api-access-w8j69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.545994 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.546059 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token" pod "ede56218-956c-417d-a362-23f3e37f0f8d" (UID: "ede56218-956c-417d-a362-23f3e37f0f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.546107 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.547901 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.547856 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token" pod "c2d88b32-d6e3-4077-a805-2d88867c93ad" (UID: "c2d88b32-d6e3-4077-a805-2d88867c93ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:18.560315 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.560227 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:59:18.632942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.632906 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.632968 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633027 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633054 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633139 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633274 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.633334 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633323 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633388 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633501 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633426 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633612 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633515 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket\") pod \"4297f2be-be32-4a9a-bd71-c1d294e1f569\" (UID: \"4297f2be-be32-4a9a-bd71-c1d294e1f569\") " Nov 03 20:59:18.633612 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.633565 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.634098 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634068 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634117 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634136 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2d88b32-d6e3-4077-a805-2d88867c93ad-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634159 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634174 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede56218-956c-417d-a362-23f3e37f0f8d-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634188 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8j69\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-kube-api-access-w8j69\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634201 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede56218-956c-417d-a362-23f3e37f0f8d-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634220 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634214 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634227 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmng\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-kube-api-access-kxmng\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634241 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634256 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede56218-956c-417d-a362-23f3e37f0f8d-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634271 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2d88b32-d6e3-4077-a805-2d88867c93ad-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634285 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede56218-956c-417d-a362-23f3e37f0f8d-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634297 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.634507 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.634297 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.636449 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.636422 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:18.636547 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.636492 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:18.637820 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.637785 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "kube-api-access-m2mwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:18.638440 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.638415 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token" pod "4297f2be-be32-4a9a-bd71-c1d294e1f569" (UID: "4297f2be-be32-4a9a-bd71-c1d294e1f569"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736021 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2mwt\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-kube-api-access-m2mwt\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736057 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736072 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736085 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297f2be-be32-4a9a-bd71-c1d294e1f569-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736100 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736115 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297f2be-be32-4a9a-bd71-c1d294e1f569-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:18.736152 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:18.736127 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297f2be-be32-4a9a-bd71-c1d294e1f569-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:19.323477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.323440 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" Nov 03 20:59:19.324007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.323477 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx" event={"ID":"c2d88b32-d6e3-4077-a805-2d88867c93ad","Type":"ContainerDied","Data":"f82bbb008ed0af793883d83a8b1f8a2b738f353ea6c062e69acafb4657fe2392"} Nov 03 20:59:19.324007 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.323561 5932 scope.go:122] "RemoveContainer" containerID="f95d30db4e42440be84837d49ddf5fddb207ebdec337b1d1e2a043bca952698b" Nov 03 20:59:19.325218 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.325191 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" Nov 03 20:59:19.325356 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.325330 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r" event={"ID":"4297f2be-be32-4a9a-bd71-c1d294e1f569","Type":"ContainerDied","Data":"83c05b019c28f97a805583453f341f28202138af19f5dae5354109e7e82df9bc"} Nov 03 20:59:19.341963 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.341069 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" event={"ID":"9b2828a1-58e3-4cfa-8ac7-760aab663beb","Type":"ContainerStarted","Data":"0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37"} Nov 03 20:59:19.346926 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.344782 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" event={"ID":"ede56218-956c-417d-a362-23f3e37f0f8d","Type":"ContainerDied","Data":"4145eabe72da68d547ba00fade6e92b02d1e464c6ad0e5b0f83981360494944e"} Nov 03 20:59:19.346926 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.344900 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp" Nov 03 20:59:19.353637 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.351739 5932 scope.go:122] "RemoveContainer" containerID="974401bb5ad969553fc0e1b40a98f4aa6e0f22edca768219849baa9fe4784c49" Nov 03 20:59:19.371407 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.371375 5932 scope.go:122] "RemoveContainer" containerID="d81dfab08b900af97b81a1bf0526262460075f2a7e2575451ac4f44979e2a1e3" Nov 03 20:59:19.606650 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.606449 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r"] Nov 03 20:59:19.610781 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.610753 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-bx67r"] Nov 03 20:59:19.747078 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.741173 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx"] Nov 03 20:59:19.751285 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.751247 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-8w8zx"] Nov 03 20:59:19.855327 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.855296 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp"] Nov 03 20:59:19.862471 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:19.862401 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-64xvp"] Nov 03 20:59:20.065111 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:20.065070 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" path="/var/lib/kubelet/pods/4297f2be-be32-4a9a-bd71-c1d294e1f569/volumes" Nov 03 20:59:20.065574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:20.065559 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" path="/var/lib/kubelet/pods/c2d88b32-d6e3-4077-a805-2d88867c93ad/volumes" Nov 03 20:59:20.066031 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:20.066017 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" path="/var/lib/kubelet/pods/ede56218-956c-417d-a362-23f3e37f0f8d/volumes" Nov 03 20:59:20.113732 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:20.113671 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:59:20.114644 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:20.114609 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:59:20.114776 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:20.114666 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:59:21.113729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:21.113663 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:59:21.113729 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:21.113724 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:59:21.136648 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:21.126705 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:59:22.114134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:22.114101 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:59:22.114580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:22.114162 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:59:23.113730 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:23.113695 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:59:23.113918 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:23.113753 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:59:24.114389 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:24.114349 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:59:24.114863 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:24.114408 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 20:59:25.118733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:25.118699 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:59:25.119718 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:25.119699 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 20:59:27.968018 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:27.967981 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm"] Nov 03 20:59:27.968436 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:27.968195 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" containerID="cri-o://de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b" gracePeriod=30 Nov 03 20:59:28.014652 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:28.014607 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n"] Nov 03 20:59:28.015134 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:28.015098 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" containerID="cri-o://b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0" gracePeriod=30 Nov 03 20:59:28.020535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:28.020504 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk"] Nov 03 20:59:28.036848 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:28.036802 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" containerID="cri-o://56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb" gracePeriod=30 Nov 03 20:59:28.073349 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:28.073298 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk"] Nov 03 20:59:28.073825 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:28.073784 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" containerID="cri-o://72b9378a0d0994215aa306c4f235090f1bad40340b1532c360edf4cedc09a58a" gracePeriod=30 Nov 03 20:59:32.220179 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.220134 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:59:32.220573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.220219 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:59:32.338699 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.338654 5932 patch_prober.go:29] interesting pod/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:59:32.338866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.338730 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:59:32.435196 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.435160 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk"] Nov 03 20:59:32.436216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436191 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" Nov 03 20:59:32.436216 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436213 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436247 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436255 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436302 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436310 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436345 5932 cpu_manager.go:414] "containerMap: removing container" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436351 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436370 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" Nov 03 20:59:32.436415 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436378 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" Nov 03 20:59:32.436840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436458 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" Nov 03 20:59:32.436840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436468 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" Nov 03 20:59:32.436840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436483 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" Nov 03 20:59:32.436840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436491 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" Nov 03 20:59:32.436840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436811 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd9ee5ad-f460-40e9-9a88-721b89da9f86" containerName="istio-proxy" Nov 03 20:59:32.436840 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436838 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede56218-956c-417d-a362-23f3e37f0f8d" containerName="istio-proxy" Nov 03 20:59:32.437138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436850 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d88b32-d6e3-4077-a805-2d88867c93ad" containerName="istio-proxy" Nov 03 20:59:32.437138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436858 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4297f2be-be32-4a9a-bd71-c1d294e1f569" containerName="istio-proxy" Nov 03 20:59:32.437138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436873 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d3e97ff-af2c-4099-94b1-e5d45b325b2d" containerName="istio-proxy" Nov 03 20:59:32.437138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436887 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0782bf0-b851-4167-ab65-38ba0be007d2" containerName="istio-proxy" Nov 03 20:59:32.437138 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.436905 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d854bc-b23f-4902-9985-ded9777dcf0a" containerName="istio-proxy" Nov 03 20:59:32.442824 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.442801 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.498022 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.497947 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk"] Nov 03 20:59:32.534509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.534473 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e68f8112\"/\"default-dockercfg-gqck6\"" Nov 03 20:59:32.534509 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.534477 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e68f8112\"/\"e2e-lifecycle-model-e68f8112-kserve-self-signed-certs\"" Nov 03 20:59:32.534762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.534481 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e68f8112\"/\"kube-root-ca.crt\"" Nov 03 20:59:32.534762 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.534487 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e68f8112\"/\"openshift-service-ca.crt\"" Nov 03 20:59:32.615646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.614860 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.615646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.614917 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.615646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.614970 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.615646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.615090 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.615646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.615154 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.615646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.615210 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717323 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717287 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717336 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717414 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717531 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717591 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717801 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717691 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.717926 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.717798 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.718092 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.718064 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.718555 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.718531 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.720768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.720742 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.721883 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.720848 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:32.776448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:32.776370 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd\") pod \"e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:33.088645 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.065858 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:33.427427 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.427404 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:59:33.517788 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.516967 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk"] Nov 03 20:59:33.524778 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:59:33.523220 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafb3b94d_a194_410a_aecf_9dfbff512cd2.slice/crio-82a0614e672e538e6295635f00d9a093927037d0ad0741f159f0aafbbc4337ff WatchSource:0}: Error finding container 82a0614e672e538e6295635f00d9a093927037d0ad0741f159f0aafbbc4337ff: Status 404 returned error can't find the container with id 82a0614e672e538e6295635f00d9a093927037d0ad0741f159f0aafbbc4337ff Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568527 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568614 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568697 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568736 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568834 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568941 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.568978 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.569063 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.569601 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.569104 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token\") pod \"376ffb76-c668-4150-b049-14ef5689f951\" (UID: \"376ffb76-c668-4150-b049-14ef5689f951\") " Nov 03 20:59:33.572362 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.571857 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:59:33.577768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.576318 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.577768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.576411 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.577768 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.577456 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.578122 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.577774 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.578845 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.578798 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.580258 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.580207 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.582965 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.582927 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.586434 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.586411 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "kube-api-access-qxnkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.586534 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.586448 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token" pod "376ffb76-c668-4150-b049-14ef5689f951" (UID: "376ffb76-c668-4150-b049-14ef5689f951"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.591339 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.591225 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:59:33.591434 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.591407 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" event={"ID":"afb3b94d-a194-410a-aecf-9dfbff512cd2","Type":"ContainerStarted","Data":"82a0614e672e538e6295635f00d9a093927037d0ad0741f159f0aafbbc4337ff"} Nov 03 20:59:33.595084 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.595034 5932 generic.go:356] "Generic (PLEG): container finished" podID="d573ec1d-a30a-4188-834c-679db4e61a98" containerID="72b9378a0d0994215aa306c4f235090f1bad40340b1532c360edf4cedc09a58a" exitCode=0 Nov 03 20:59:33.595191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.595124 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" event={"ID":"d573ec1d-a30a-4188-834c-679db4e61a98","Type":"ContainerDied","Data":"72b9378a0d0994215aa306c4f235090f1bad40340b1532c360edf4cedc09a58a"} Nov 03 20:59:33.597119 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.597093 5932 generic.go:356] "Generic (PLEG): container finished" podID="376ffb76-c668-4150-b049-14ef5689f951" containerID="de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b" exitCode=0 Nov 03 20:59:33.597225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.597175 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" event={"ID":"376ffb76-c668-4150-b049-14ef5689f951","Type":"ContainerDied","Data":"de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b"} Nov 03 20:59:33.597225 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.597216 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" event={"ID":"376ffb76-c668-4150-b049-14ef5689f951","Type":"ContainerDied","Data":"c454801a54a07b27264bcc4b4bad200564d31bd7010d00548c785d2bed57ba75"} Nov 03 20:59:33.597343 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.597241 5932 scope.go:122] "RemoveContainer" containerID="de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b" Nov 03 20:59:33.597404 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.597355 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm" Nov 03 20:59:33.602529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.601991 5932 generic.go:356] "Generic (PLEG): container finished" podID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerID="56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb" exitCode=0 Nov 03 20:59:33.602529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.602159 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" event={"ID":"3e16e9c5-c36e-4026-8e0e-775c6329f41d","Type":"ContainerDied","Data":"56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb"} Nov 03 20:59:33.602529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.602202 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" event={"ID":"3e16e9c5-c36e-4026-8e0e-775c6329f41d","Type":"ContainerDied","Data":"1971e3841afbb863bcc41c6c0ed4adbe358a6c1cb9f82798b675cf12f81fb947"} Nov 03 20:59:33.602529 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.602274 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk" Nov 03 20:59:33.604112 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.604089 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:59:33.604745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.604454 5932 generic.go:356] "Generic (PLEG): container finished" podID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerID="b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0" exitCode=0 Nov 03 20:59:33.604745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.604487 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" event={"ID":"fff55790-9ea4-4411-a3b7-6e1c5eabe989","Type":"ContainerDied","Data":"b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0"} Nov 03 20:59:33.604745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.604523 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" event={"ID":"fff55790-9ea4-4411-a3b7-6e1c5eabe989","Type":"ContainerDied","Data":"baec34eb639dd38a3eb722d3c6470eeae8cb95f7f6efa4caa7da25d1e2c9c3d9"} Nov 03 20:59:33.604745 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.604590 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n" Nov 03 20:59:33.615319 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.615299 5932 scope.go:122] "RemoveContainer" containerID="de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b" Nov 03 20:59:33.615713 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:33.615688 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b\": container with ID starting with de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b not found: ID does not exist" containerID="de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b" Nov 03 20:59:33.615818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.615723 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b"} err="failed to get container status \"de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b\": rpc error: code = NotFound desc = could not find container \"de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b\": container with ID starting with de06ca6420363f29237b9309bc02ba1490a7f607cd59adf6ce4a42175e58960b not found: ID does not exist" Nov 03 20:59:33.615818 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.615743 5932 scope.go:122] "RemoveContainer" containerID="56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb" Nov 03 20:59:33.630211 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.630194 5932 scope.go:122] "RemoveContainer" containerID="56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb" Nov 03 20:59:33.630509 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:33.630487 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb\": container with ID starting with 56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb not found: ID does not exist" containerID="56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb" Nov 03 20:59:33.630595 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.630519 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb"} err="failed to get container status \"56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb\": rpc error: code = NotFound desc = could not find container \"56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb\": container with ID starting with 56aa277a8d0d501792ab5ee0dca639a002a76461ea2dcc3079817d7a3fbfc5bb not found: ID does not exist" Nov 03 20:59:33.630595 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.630541 5932 scope.go:122] "RemoveContainer" containerID="b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0" Nov 03 20:59:33.646448 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.646423 5932 scope.go:122] "RemoveContainer" containerID="b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0" Nov 03 20:59:33.646807 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:33.646783 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0\": container with ID starting with b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0 not found: ID does not exist" containerID="b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0" Nov 03 20:59:33.646918 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.646813 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0"} err="failed to get container status \"b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0\": rpc error: code = NotFound desc = could not find container \"b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0\": container with ID starting with b26ca7fad4fb9457fa6f35715d0f0f7d93d352f3ddb17fa881f1d572537bdea0 not found: ID does not exist" Nov 03 20:59:33.677050 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677016 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.677199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677076 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.677199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677113 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.677199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677148 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.677199 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677183 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.677403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677228 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.677403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677339 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.677403 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677388 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.677544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677481 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.677544 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677511 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.677733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677551 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.677733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677577 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.677733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677605 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.677733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677649 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.677733 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677694 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.678039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677739 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.678039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677819 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.678039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677871 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.678039 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.677937 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.678249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678054 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.678249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678093 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.678249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678152 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.678249 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678219 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token\") pod \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\" (UID: \"fff55790-9ea4-4411-a3b7-6e1c5eabe989\") " Nov 03 20:59:33.678466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678258 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.678466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678297 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.678466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678345 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy\") pod \"d573ec1d-a30a-4188-834c-679db4e61a98\" (UID: \"d573ec1d-a30a-4188-834c-679db4e61a98\") " Nov 03 20:59:33.678466 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.678388 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token\") pod \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\" (UID: \"3e16e9c5-c36e-4026-8e0e-775c6329f41d\") " Nov 03 20:59:33.679398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.679363 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.679398 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.679388 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.679866 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.679843 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.680525 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680500 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnkr\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-kube-api-access-qxnkr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680529 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680547 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680562 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680576 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/376ffb76-c668-4150-b049-14ef5689f951-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680589 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680602 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680616 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680644 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680659 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/376ffb76-c668-4150-b049-14ef5689f951-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680672 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/376ffb76-c668-4150-b049-14ef5689f951-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.680706 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680686 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/376ffb76-c668-4150-b049-14ef5689f951-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.681275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.680923 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.681275 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.681209 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.684597 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.683591 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.684720 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.684613 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.685150 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.684873 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.686373 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.686347 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.688136 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.689253 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.689439 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.689608 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.689807 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.689996 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.691687 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.690176 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.692296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.692117 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.692296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.692273 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.692429 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.692406 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:59:33.694433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.693696 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "kube-api-access-l68gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.694433 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.694345 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:59:33.694853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.694775 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.694853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.694777 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.694853 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.694790 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:59:33.695043 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.694865 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm" pod "d573ec1d-a30a-4188-834c-679db4e61a98" (UID: "d573ec1d-a30a-4188-834c-679db4e61a98"). InnerVolumeSpecName "kube-api-access-s6hmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.699908 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.699879 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn" pod "3e16e9c5-c36e-4026-8e0e-775c6329f41d" (UID: "3e16e9c5-c36e-4026-8e0e-775c6329f41d"). InnerVolumeSpecName "kube-api-access-nwhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.700082 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.700060 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token" pod "fff55790-9ea4-4411-a3b7-6e1c5eabe989" (UID: "fff55790-9ea4-4411-a3b7-6e1c5eabe989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:59:33.782106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782073 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782106 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782103 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d573ec1d-a30a-4188-834c-679db4e61a98-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782118 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782131 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782144 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782155 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d573ec1d-a30a-4188-834c-679db4e61a98-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782168 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6hmm\" (UniqueName: \"kubernetes.io/projected/d573ec1d-a30a-4188-834c-679db4e61a98-kube-api-access-s6hmm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782181 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782195 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782206 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782218 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782231 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782243 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782254 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782268 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68gt\" (UniqueName: \"kubernetes.io/projected/fff55790-9ea4-4411-a3b7-6e1c5eabe989-kube-api-access-l68gt\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782282 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e16e9c5-c36e-4026-8e0e-775c6329f41d-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782296 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782316 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782309 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhcn\" (UniqueName: \"kubernetes.io/projected/3e16e9c5-c36e-4026-8e0e-775c6329f41d-kube-api-access-nwhcn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782324 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782339 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782354 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782367 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff55790-9ea4-4411-a3b7-6e1c5eabe989-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782381 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d573ec1d-a30a-4188-834c-679db4e61a98-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:33.782922 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:33.782395 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e16e9c5-c36e-4026-8e0e-775c6329f41d-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 20:59:34.077485 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.077454 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm"] Nov 03 20:59:34.088307 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.084797 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-5rjsm"] Nov 03 20:59:34.186477 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.186398 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n"] Nov 03 20:59:34.192164 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.192139 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-zdw7n"] Nov 03 20:59:34.297564 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.297530 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk"] Nov 03 20:59:34.301573 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.301548 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-9sjzk"] Nov 03 20:59:34.617013 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.616924 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" event={"ID":"afb3b94d-a194-410a-aecf-9dfbff512cd2","Type":"ContainerStarted","Data":"53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f"} Nov 03 20:59:34.618482 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.618454 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" Nov 03 20:59:34.618755 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.618461 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk" event={"ID":"d573ec1d-a30a-4188-834c-679db4e61a98","Type":"ContainerDied","Data":"f8172c2fc186b9cfae9ad73c94c2c4101a71c4e02b78855b39f2493a168bc516"} Nov 03 20:59:34.619350 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.619328 5932 scope.go:122] "RemoveContainer" containerID="72b9378a0d0994215aa306c4f235090f1bad40340b1532c360edf4cedc09a58a" Nov 03 20:59:34.731191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.731140 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podStartSLOduration=2.731126039 podStartE2EDuration="2.731126039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:34.729469832 +0000 UTC m=+1782.794711307" watchObservedRunningTime="2026-11-03 20:59:34.731126039 +0000 UTC m=+1782.796367514" Nov 03 20:59:34.892747 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.892700 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk"] Nov 03 20:59:34.906810 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:34.905911 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1e6fe1-openshift-default-7d5d6c5549-wmjnk"] Nov 03 20:59:35.067157 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:35.067123 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:36.053497 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.053468 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="376ffb76-c668-4150-b049-14ef5689f951" path="/var/lib/kubelet/pods/376ffb76-c668-4150-b049-14ef5689f951/volumes" Nov 03 20:59:36.054033 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.054015 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" path="/var/lib/kubelet/pods/3e16e9c5-c36e-4026-8e0e-775c6329f41d/volumes" Nov 03 20:59:36.054491 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.054474 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" path="/var/lib/kubelet/pods/d573ec1d-a30a-4188-834c-679db4e61a98/volumes" Nov 03 20:59:36.055670 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.055650 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" path="/var/lib/kubelet/pods/fff55790-9ea4-4411-a3b7-6e1c5eabe989/volumes" Nov 03 20:59:36.647678 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.647610 5932 generic.go:356] "Generic (PLEG): container finished" podID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerID="86dcd017b8e1a7601353d63cae759d39d27064a7f8f643653073c259b0150647" exitCode=0 Nov 03 20:59:36.647869 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.647652 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" event={"ID":"03d0cb91-0699-494a-9595-7d06a9c49e60","Type":"ContainerDied","Data":"86dcd017b8e1a7601353d63cae759d39d27064a7f8f643653073c259b0150647"} Nov 03 20:59:36.648191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.648170 5932 scope.go:122] "RemoveContainer" containerID="86dcd017b8e1a7601353d63cae759d39d27064a7f8f643653073c259b0150647" Nov 03 20:59:36.649749 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.649717 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:36.649826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.649788 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:36.649826 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:36.649819 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 20:59:37.658535 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:37.658500 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" event={"ID":"03d0cb91-0699-494a-9595-7d06a9c49e60","Type":"ContainerStarted","Data":"a4e1e79577e1ee7fe5bea83e9427c59a6acbab6d246b650d1e15d99a182d890a"} Nov 03 20:59:37.686453 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:37.686424 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:59:37.687480 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:37.687456 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:37.687580 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:37.687501 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:38.687363 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:38.687331 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:38.687767 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:38.687383 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:39.100508 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:39.100437 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:59:39.686564 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:39.686529 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:39.686780 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:39.686587 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:40.686434 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:40.686398 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:40.686892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:40.686457 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:41.686710 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:41.686672 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:41.687087 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:41.686747 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:42.687229 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:42.687196 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:42.687641 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:42.687250 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:43.066950 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:43.066858 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:43.080588 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:43.080558 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:43.687286 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:43.687250 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:59:43.687743 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:43.687322 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 20:59:43.747920 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:43.747882 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 20:59:44.152922 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:44.152879 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 20:59:44.729646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:44.721038 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:59:44.738646 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:44.737320 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 20:59:50.263261 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.263225 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9"] Nov 03 20:59:50.264008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.263992 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" Nov 03 20:59:50.264008 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264008 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" Nov 03 20:59:50.264109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264022 5932 cpu_manager.go:414] "containerMap: removing container" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" Nov 03 20:59:50.264109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264027 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" Nov 03 20:59:50.264109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264049 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" Nov 03 20:59:50.264109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264054 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" Nov 03 20:59:50.264109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264060 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" Nov 03 20:59:50.264109 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264065 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" Nov 03 20:59:50.264293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264161 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff55790-9ea4-4411-a3b7-6e1c5eabe989" containerName="istio-proxy" Nov 03 20:59:50.264293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264170 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e16e9c5-c36e-4026-8e0e-775c6329f41d" containerName="istio-proxy" Nov 03 20:59:50.264293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264185 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d573ec1d-a30a-4188-834c-679db4e61a98" containerName="istio-proxy" Nov 03 20:59:50.264293 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.264191 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="376ffb76-c668-4150-b049-14ef5689f951" containerName="istio-proxy" Nov 03 20:59:50.268028 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.268011 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.321783 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.321748 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9"] Nov 03 20:59:50.356671 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.356601 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.356838 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.356763 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wrkk\" (UniqueName: \"kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.356892 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.356861 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.356970 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.356957 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.362942 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.362913 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2e3ea2-serving-cert\"" Nov 03 20:59:50.363075 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.363012 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:59:50.363139 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.363094 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\"" Nov 03 20:59:50.458223 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.458186 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.458383 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.458284 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wrkk\" (UniqueName: \"kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.458430 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.458385 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.458505 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.458490 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.458546 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:50.458521 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-serving-cert: secret "maas-api-e2e-shared-a-w3-2e3ea2-serving-cert" not found Nov 03 20:59:50.458612 ip-10-0-138-52 kubenswrapper[5932]: E1103 20:59:50.458596 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls podName:fa645b1b-2011-4d56-9542-407a948d3c55 nodeName:}" failed. No retries permitted until 2026-11-03 20:59:50.958576362 +0000 UTC m=+1799.023817821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" (UID: "fa645b1b-2011-4d56-9542-407a948d3c55") : secret "maas-api-e2e-shared-a-w3-2e3ea2-serving-cert" not found Nov 03 20:59:50.459243 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.459217 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.460792 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.460770 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.528574 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.527703 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wrkk\" (UniqueName: \"kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.972356 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.972321 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:50.974937 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:50.974911 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:51.182895 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:51.182862 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:51.567097 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:51.567010 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9"] Nov 03 20:59:51.572081 ip-10-0-138-52 kubenswrapper[5932]: W1103 20:59:51.572055 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa645b1b_2011_4d56_9542_407a948d3c55.slice/crio-64c3013fd387f7fb5d06ab6854394fa4737cb734d8c464fc3b3119c2c191fd4f WatchSource:0}: Error finding container 64c3013fd387f7fb5d06ab6854394fa4737cb734d8c464fc3b3119c2c191fd4f: Status 404 returned error can't find the container with id 64c3013fd387f7fb5d06ab6854394fa4737cb734d8c464fc3b3119c2c191fd4f Nov 03 20:59:51.573722 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:51.573699 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:59:51.976198 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:51.976161 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" event={"ID":"fa645b1b-2011-4d56-9542-407a948d3c55","Type":"ContainerStarted","Data":"64c3013fd387f7fb5d06ab6854394fa4737cb734d8c464fc3b3119c2c191fd4f"} Nov 03 20:59:55.007048 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.007004 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" event={"ID":"fa645b1b-2011-4d56-9542-407a948d3c55","Type":"ContainerStarted","Data":"5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f"} Nov 03 20:59:55.007481 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.007071 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 20:59:55.124473 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.124409 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" podStartSLOduration=2.3685954860000002 podStartE2EDuration="5.124390946s" totalImagesPullingTime="2.75579546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:59:55.117483881 +0000 UTC m=+1803.182725355" watchObservedRunningTime="2026-11-03 20:59:55.124390946 +0000 UTC m=+1803.189632419" Nov 03 20:59:55.665659 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.663220 5932 scope.go:122] "RemoveContainer" containerID="e8c405912afdb0099b1c030a84a4d602399eb8eabc49ea7835ea811bb7bc1f65" Nov 03 20:59:55.684214 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.684186 5932 scope.go:122] "RemoveContainer" containerID="554a1c43d87c2501dc0007e2259816c9b68cdd8ae1f7e1f0c62b728bba99854c" Nov 03 20:59:55.694191 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.694170 5932 scope.go:122] "RemoveContainer" containerID="93675c229ae9fce7cb7e51f0222d2716122dff15ec18bf5c31b5fd9264a1cef7" Nov 03 20:59:55.704615 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.704592 5932 scope.go:122] "RemoveContainer" containerID="e923ef7d29d869b36a6fa382871d9e1aacbbfabd6b3a80f2697f99b108bcd069" Nov 03 20:59:55.953296 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.953212 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 20:59:55.964046 ip-10-0-138-52 kubenswrapper[5932]: I1103 20:59:55.964023 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:00:00.170798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.170762 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms"] Nov 03 21:00:00.177765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.177740 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.178497 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.178473 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz"] Nov 03 21:00:00.183838 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.183817 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r"] Nov 03 21:00:00.183969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.183951 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.188695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.188674 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.216909 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.216885 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq"] Nov 03 21:00:00.221247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.221227 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz"] Nov 03 21:00:00.221365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.221348 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.226786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.226763 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.235490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.235469 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd"] Nov 03 21:00:00.240321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.240298 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64"] Nov 03 21:00:00.240429 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.240409 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.244809 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.244786 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd"] Nov 03 21:00:00.244915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.244891 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.249551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.249530 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.259006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.258985 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:00:00.267055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.267033 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz"] Nov 03 21:00:00.287673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.286476 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms"] Nov 03 21:00:00.288819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.288794 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh"] Nov 03 21:00:00.295320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.295297 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.299839 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.299812 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r"] Nov 03 21:00:00.305158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.304779 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq"] Nov 03 21:00:00.336101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.336069 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz"] Nov 03 21:00:00.339899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.339863 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q7kx\" (UniqueName: \"kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.340090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340060 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdvlq\" (UniqueName: \"kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.340193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340104 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj5s\" (UniqueName: \"kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.340193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340163 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.340304 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340192 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.340304 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340269 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.340411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340352 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.340463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340437 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.340519 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340469 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.340519 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340509 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqfsp\" (UniqueName: \"kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.340643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340560 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtsl\" (UniqueName: \"kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.340643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340600 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p758h\" (UniqueName: \"kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h\") pod \"maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.340757 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340684 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.340757 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340742 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.340864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340775 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86d8\" (UniqueName: \"kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.340922 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.340869 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz4jm\" (UniqueName: \"kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm\") pod \"maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.349590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.349555 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64"] Nov 03 21:00:00.356182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.356160 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd"] Nov 03 21:00:00.361318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.361283 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh"] Nov 03 21:00:00.372236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.372207 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd"] Nov 03 21:00:00.441947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.441848 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p758h\" (UniqueName: \"kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h\") pod \"maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.441947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.441896 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zff\" (UniqueName: \"kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.442205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442032 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.442205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442126 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.442205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442158 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86d8\" (UniqueName: \"kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.442459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442335 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz4jm\" (UniqueName: \"kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm\") pod \"maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.442459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442439 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q7kx\" (UniqueName: \"kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.442589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442472 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdvlq\" (UniqueName: \"kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.442589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442503 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj5s\" (UniqueName: \"kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.442589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442564 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.442906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442603 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.442906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442738 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.442906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442878 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.443073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.442959 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.443073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443030 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.443186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443115 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.443186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443126 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.443186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443171 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.443385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443211 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqfsp\" (UniqueName: \"kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.443385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443267 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtsl\" (UniqueName: \"kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.443870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443844 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.443983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443902 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.443983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.443950 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.444117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.444024 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.444207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.444186 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.444287 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.444271 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.500453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.500411 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q7kx\" (UniqueName: \"kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.500687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.500644 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz4jm\" (UniqueName: \"kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm\") pod \"maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.500840 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.500818 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdvlq\" (UniqueName: \"kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.500910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.500821 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p758h\" (UniqueName: \"kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h\") pod \"maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.500910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.500891 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtsl\" (UniqueName: \"kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.501015 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.500946 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj5s\" (UniqueName: \"kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s\") pod \"maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.501124 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.501102 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86d8\" (UniqueName: \"kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8\") pod \"maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.502989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.502966 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqfsp\" (UniqueName: \"kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.503796 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.503780 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:00.512379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.512360 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:00.540806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.540780 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:00.544370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.544332 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.544536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.544515 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zff\" (UniqueName: \"kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.546005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.545252 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.551391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.551367 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:00.560373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.560163 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:00.570182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.569989 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:00.582007 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.581489 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:00.618337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.618269 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zff\" (UniqueName: \"kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.627651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.627413 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:00.806252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.803496 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:00.907414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:00.907379 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz"] Nov 03 21:00:00.909581 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:00.909551 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a0522c_60ab_4663_9f9a_bf8179386990.slice/crio-0f90f105a8c890e908c0850e19a8f46666c927a5a64e45ff2ba0e95f4595695a WatchSource:0}: Error finding container 0f90f105a8c890e908c0850e19a8f46666c927a5a64e45ff2ba0e95f4595695a: Status 404 returned error can't find the container with id 0f90f105a8c890e908c0850e19a8f46666c927a5a64e45ff2ba0e95f4595695a Nov 03 21:00:01.024488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.024414 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 21:00:01.079237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.079196 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerStarted","Data":"0f90f105a8c890e908c0850e19a8f46666c927a5a64e45ff2ba0e95f4595695a"} Nov 03 21:00:01.111276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.111247 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd"] Nov 03 21:00:01.116974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.116949 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r"] Nov 03 21:00:01.124985 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:01.124958 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1decd89c_8bce_4bd9_b698_68910d386e95.slice/crio-529e70e3c15ac33270cd3f82c8440d8ba73eb96505deb5ddc28289604695f684 WatchSource:0}: Error finding container 529e70e3c15ac33270cd3f82c8440d8ba73eb96505deb5ddc28289604695f684: Status 404 returned error can't find the container with id 529e70e3c15ac33270cd3f82c8440d8ba73eb96505deb5ddc28289604695f684 Nov 03 21:00:01.125768 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:01.125739 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba61f92_9dde_40ea_baa8_954bcb70c415.slice/crio-d98fc86b5516cb3b8fa2538129341a3a9384292f7c25999842136cbeb2ab4f5b WatchSource:0}: Error finding container d98fc86b5516cb3b8fa2538129341a3a9384292f7c25999842136cbeb2ab4f5b: Status 404 returned error can't find the container with id d98fc86b5516cb3b8fa2538129341a3a9384292f7c25999842136cbeb2ab4f5b Nov 03 21:00:01.176167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.176140 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq"] Nov 03 21:00:01.178866 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:01.178829 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ccfc2_5101_4881_9340_7d4f701c1b01.slice/crio-53b7998aa4b4d7edb68fb69c8a6c3664b8df4797dd7b78ae144bc8359385ff25 WatchSource:0}: Error finding container 53b7998aa4b4d7edb68fb69c8a6c3664b8df4797dd7b78ae144bc8359385ff25: Status 404 returned error can't find the container with id 53b7998aa4b4d7edb68fb69c8a6c3664b8df4797dd7b78ae144bc8359385ff25 Nov 03 21:00:01.216556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.216526 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms"] Nov 03 21:00:01.240273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.240246 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz"] Nov 03 21:00:01.312983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.312688 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh"] Nov 03 21:00:01.321549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.321518 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd"] Nov 03 21:00:01.335548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:01.334616 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64"] Nov 03 21:00:01.439207 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:01.439179 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210ea6fe_9453_40e0_8fbe_22eef3f023a5.slice/crio-beaca0764e24ca4368cd74478571abc4ec1a3706dc959ee458b902ef9799c60e WatchSource:0}: Error finding container beaca0764e24ca4368cd74478571abc4ec1a3706dc959ee458b902ef9799c60e: Status 404 returned error can't find the container with id beaca0764e24ca4368cd74478571abc4ec1a3706dc959ee458b902ef9799c60e Nov 03 21:00:02.094226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.094186 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerStarted","Data":"0831848b1758fd4f14e5953f3707628233b3ecbbae5dbcc271d3a639465dd101"} Nov 03 21:00:02.094469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.094275 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerStarted","Data":"4ce5f4d4a525e1a205db2cb45fad2d3cf206b40033db3e3a8c563ef0b71c71f7"} Nov 03 21:00:02.095885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.095854 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerStarted","Data":"e33bbfdd776d47d229e94ff00b5a1ba1e7941fe88c14ecbd0818fdd86ff4b7be"} Nov 03 21:00:02.096016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.095911 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerStarted","Data":"53b7998aa4b4d7edb68fb69c8a6c3664b8df4797dd7b78ae144bc8359385ff25"} Nov 03 21:00:02.097208 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.097181 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerStarted","Data":"583318c01e5a367697bb25f1bb61c607e4aaedd3159d8c28493e814a8de5afc3"} Nov 03 21:00:02.098892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.098868 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerStarted","Data":"f55944d18c8da1a0e302aced866e37af1b9b284e38cf0343decd453164e5ca6b"} Nov 03 21:00:02.098993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.098916 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerStarted","Data":"529e70e3c15ac33270cd3f82c8440d8ba73eb96505deb5ddc28289604695f684"} Nov 03 21:00:02.100414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.100390 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerStarted","Data":"95d7e404b95da2837dffe6b7a0be1447dea1274c5f7b44b3ae86c89054534646"} Nov 03 21:00:02.100515 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.100430 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerStarted","Data":"d98fc86b5516cb3b8fa2538129341a3a9384292f7c25999842136cbeb2ab4f5b"} Nov 03 21:00:02.101712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.101690 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerStarted","Data":"49cd16d3555cff4d00fcf7e0011ddac3ce10afa57e4760c2963d5151935d1ba7"} Nov 03 21:00:02.102813 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.102790 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerStarted","Data":"2fade6a785696f407f216a659ea9d62be2f15eccf6d1e03cd50f53fab1da6e2d"} Nov 03 21:00:02.104301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.104279 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerStarted","Data":"019671d5abcda3e528994128a986ffd9058511dd01ecae2063d50ff8e31badae"} Nov 03 21:00:02.105524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.105504 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerStarted","Data":"beaca0764e24ca4368cd74478571abc4ec1a3706dc959ee458b902ef9799c60e"} Nov 03 21:00:02.206992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.206904 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" podStartSLOduration=2.206889453 podStartE2EDuration="2.206889453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.205896157 +0000 UTC m=+1810.271137631" watchObservedRunningTime="2026-11-03 21:00:02.206889453 +0000 UTC m=+1810.272130939" Nov 03 21:00:02.316227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.316169 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" podStartSLOduration=2.316154978 podStartE2EDuration="2.316154978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.313282678 +0000 UTC m=+1810.378524151" watchObservedRunningTime="2026-11-03 21:00:02.316154978 +0000 UTC m=+1810.381396445" Nov 03 21:00:02.427299 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.423836 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" podStartSLOduration=2.423818169 podStartE2EDuration="2.423818169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.420774 +0000 UTC m=+1810.486015474" watchObservedRunningTime="2026-11-03 21:00:02.423818169 +0000 UTC m=+1810.489059643" Nov 03 21:00:02.534442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.534384 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" podStartSLOduration=2.5343674309999997 podStartE2EDuration="2.534367431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.528501046 +0000 UTC m=+1810.593742518" watchObservedRunningTime="2026-11-03 21:00:02.534367431 +0000 UTC m=+1810.599608905" Nov 03 21:00:02.637441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:02.637388 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" podStartSLOduration=2.6373741 podStartE2EDuration="2.6373741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:02.636091138 +0000 UTC m=+1810.701332611" watchObservedRunningTime="2026-11-03 21:00:02.6373741 +0000 UTC m=+1810.702615572" Nov 03 21:00:03.115681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.115648 5932 generic.go:356] "Generic (PLEG): container finished" podID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerID="0831848b1758fd4f14e5953f3707628233b3ecbbae5dbcc271d3a639465dd101" exitCode=60 Nov 03 21:00:03.115841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.115719 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerDied","Data":"0831848b1758fd4f14e5953f3707628233b3ecbbae5dbcc271d3a639465dd101"} Nov 03 21:00:03.116072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.116055 5932 scope.go:122] "RemoveContainer" containerID="0831848b1758fd4f14e5953f3707628233b3ecbbae5dbcc271d3a639465dd101" Nov 03 21:00:03.117473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.117442 5932 generic.go:356] "Generic (PLEG): container finished" podID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerID="e33bbfdd776d47d229e94ff00b5a1ba1e7941fe88c14ecbd0818fdd86ff4b7be" exitCode=60 Nov 03 21:00:03.117575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.117519 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerDied","Data":"e33bbfdd776d47d229e94ff00b5a1ba1e7941fe88c14ecbd0818fdd86ff4b7be"} Nov 03 21:00:03.117882 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.117808 5932 scope.go:122] "RemoveContainer" containerID="e33bbfdd776d47d229e94ff00b5a1ba1e7941fe88c14ecbd0818fdd86ff4b7be" Nov 03 21:00:03.119512 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.119488 5932 generic.go:356] "Generic (PLEG): container finished" podID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerID="a40e208538dfe28650c6f95eff5b35cdf021031c4a1b459bf8b6aeb3bb321806" exitCode=60 Nov 03 21:00:03.119710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.119652 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerDied","Data":"a40e208538dfe28650c6f95eff5b35cdf021031c4a1b459bf8b6aeb3bb321806"} Nov 03 21:00:03.119881 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.119837 5932 scope.go:122] "RemoveContainer" containerID="a40e208538dfe28650c6f95eff5b35cdf021031c4a1b459bf8b6aeb3bb321806" Nov 03 21:00:03.121970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.121951 5932 generic.go:356] "Generic (PLEG): container finished" podID="1decd89c-8bce-4bd9-b698-68910d386e95" containerID="f55944d18c8da1a0e302aced866e37af1b9b284e38cf0343decd453164e5ca6b" exitCode=60 Nov 03 21:00:03.122102 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.122086 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerDied","Data":"f55944d18c8da1a0e302aced866e37af1b9b284e38cf0343decd453164e5ca6b"} Nov 03 21:00:03.122373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.122356 5932 scope.go:122] "RemoveContainer" containerID="f55944d18c8da1a0e302aced866e37af1b9b284e38cf0343decd453164e5ca6b" Nov 03 21:00:03.124248 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.124230 5932 generic.go:356] "Generic (PLEG): container finished" podID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerID="95d7e404b95da2837dffe6b7a0be1447dea1274c5f7b44b3ae86c89054534646" exitCode=60 Nov 03 21:00:03.124346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.124303 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerDied","Data":"95d7e404b95da2837dffe6b7a0be1447dea1274c5f7b44b3ae86c89054534646"} Nov 03 21:00:03.124493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.124479 5932 scope.go:122] "RemoveContainer" containerID="95d7e404b95da2837dffe6b7a0be1447dea1274c5f7b44b3ae86c89054534646" Nov 03 21:00:03.125866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.125845 5932 generic.go:356] "Generic (PLEG): container finished" podID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerID="a850c2d2f78fac4ee58ae6019b566dde09a3ddc740aea93bca6b940d2296d167" exitCode=60 Nov 03 21:00:03.125953 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.125922 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerDied","Data":"a850c2d2f78fac4ee58ae6019b566dde09a3ddc740aea93bca6b940d2296d167"} Nov 03 21:00:03.126114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.126097 5932 scope.go:122] "RemoveContainer" containerID="a850c2d2f78fac4ee58ae6019b566dde09a3ddc740aea93bca6b940d2296d167" Nov 03 21:00:03.127698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.127680 5932 generic.go:356] "Generic (PLEG): container finished" podID="53138c21-35f3-44a8-a8e3-31000aa44218" containerID="7f3694a8cc5e96e92561ccba6e75f6da6a48708dccdedc82a0cdc0ec7502c910" exitCode=60 Nov 03 21:00:03.127780 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.127762 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerDied","Data":"7f3694a8cc5e96e92561ccba6e75f6da6a48708dccdedc82a0cdc0ec7502c910"} Nov 03 21:00:03.127931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.127918 5932 scope.go:122] "RemoveContainer" containerID="7f3694a8cc5e96e92561ccba6e75f6da6a48708dccdedc82a0cdc0ec7502c910" Nov 03 21:00:03.136092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.131072 5932 generic.go:356] "Generic (PLEG): container finished" podID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerID="019671d5abcda3e528994128a986ffd9058511dd01ecae2063d50ff8e31badae" exitCode=60 Nov 03 21:00:03.136092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.131390 5932 scope.go:122] "RemoveContainer" containerID="019671d5abcda3e528994128a986ffd9058511dd01ecae2063d50ff8e31badae" Nov 03 21:00:03.136092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.131724 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerDied","Data":"019671d5abcda3e528994128a986ffd9058511dd01ecae2063d50ff8e31badae"} Nov 03 21:00:03.138296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.138275 5932 generic.go:356] "Generic (PLEG): container finished" podID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerID="26422778ed9fc45b274f0fc79018ec4985966726e95ba1f4536c53059542b736" exitCode=60 Nov 03 21:00:03.138399 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.138350 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerDied","Data":"26422778ed9fc45b274f0fc79018ec4985966726e95ba1f4536c53059542b736"} Nov 03 21:00:03.138543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:03.138526 5932 scope.go:122] "RemoveContainer" containerID="26422778ed9fc45b274f0fc79018ec4985966726e95ba1f4536c53059542b736" Nov 03 21:00:04.150088 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.150050 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerStarted","Data":"c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd"} Nov 03 21:00:04.152255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.152231 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerStarted","Data":"3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef"} Nov 03 21:00:04.154156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.154134 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerStarted","Data":"c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9"} Nov 03 21:00:04.156014 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.155990 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerStarted","Data":"997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c"} Nov 03 21:00:04.157846 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.157826 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerStarted","Data":"a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda"} Nov 03 21:00:04.159655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.159600 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerStarted","Data":"1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec"} Nov 03 21:00:04.161654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.161602 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerStarted","Data":"4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf"} Nov 03 21:00:04.164140 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.164117 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerStarted","Data":"95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec"} Nov 03 21:00:04.166090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.166069 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerStarted","Data":"33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41"} Nov 03 21:00:04.704917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.704857 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" podStartSLOduration=4.704838295 podStartE2EDuration="4.704838295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:04.701370693 +0000 UTC m=+1812.766612161" watchObservedRunningTime="2026-11-03 21:00:04.704838295 +0000 UTC m=+1812.770079772" Nov 03 21:00:04.813445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.813390 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" podStartSLOduration=4.813374675 podStartE2EDuration="4.813374675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:04.808822666 +0000 UTC m=+1812.874064139" watchObservedRunningTime="2026-11-03 21:00:04.813374675 +0000 UTC m=+1812.878616149" Nov 03 21:00:04.924199 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:04.924150 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" podStartSLOduration=4.924130907 podStartE2EDuration="4.924130907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:04.923724558 +0000 UTC m=+1812.988966031" watchObservedRunningTime="2026-11-03 21:00:04.924130907 +0000 UTC m=+1812.989372410" Nov 03 21:00:05.066875 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.066776 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" podStartSLOduration=5.066762735 podStartE2EDuration="5.066762735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:05.06234935 +0000 UTC m=+1813.127590824" watchObservedRunningTime="2026-11-03 21:00:05.066762735 +0000 UTC m=+1813.132004208" Nov 03 21:00:05.187566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.187528 5932 generic.go:356] "Generic (PLEG): container finished" podID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerID="a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda" exitCode=60 Nov 03 21:00:05.189457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.188270 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerDied","Data":"a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda"} Nov 03 21:00:05.189662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.189347 5932 scope.go:122] "RemoveContainer" containerID="a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda" Nov 03 21:00:05.190414 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.189919 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r_odh-ai-gateway-infra(5ba61f92-9dde-40ea-baa8-954bcb70c415)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" Nov 03 21:00:05.190414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.190172 5932 scope.go:122] "RemoveContainer" containerID="95d7e404b95da2837dffe6b7a0be1447dea1274c5f7b44b3ae86c89054534646" Nov 03 21:00:05.194262 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.194239 5932 generic.go:356] "Generic (PLEG): container finished" podID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerID="1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec" exitCode=60 Nov 03 21:00:05.194352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.194323 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerDied","Data":"1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec"} Nov 03 21:00:05.194721 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.194700 5932 scope.go:122] "RemoveContainer" containerID="1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec" Nov 03 21:00:05.195195 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.195169 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz_odh-ai-gateway-infra(6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" Nov 03 21:00:05.196670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.196646 5932 generic.go:356] "Generic (PLEG): container finished" podID="53138c21-35f3-44a8-a8e3-31000aa44218" containerID="4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf" exitCode=60 Nov 03 21:00:05.197071 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.197054 5932 scope.go:122] "RemoveContainer" containerID="4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf" Nov 03 21:00:05.197351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.197329 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerDied","Data":"4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf"} Nov 03 21:00:05.197495 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.197361 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh_odh-ai-gateway-infra(53138c21-35f3-44a8-a8e3-31000aa44218)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" Nov 03 21:00:05.199390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.199366 5932 generic.go:356] "Generic (PLEG): container finished" podID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerID="95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec" exitCode=60 Nov 03 21:00:05.199496 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.199444 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerDied","Data":"95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec"} Nov 03 21:00:05.199700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.199682 5932 scope.go:122] "RemoveContainer" containerID="95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec" Nov 03 21:00:05.199967 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.199949 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz_odh-ai-gateway-infra(d7a0522c-60ab-4663-9f9a-bf8179386990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" Nov 03 21:00:05.201426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.201404 5932 generic.go:356] "Generic (PLEG): container finished" podID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerID="33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41" exitCode=60 Nov 03 21:00:05.201603 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.201577 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerDied","Data":"33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41"} Nov 03 21:00:05.201793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.201779 5932 scope.go:122] "RemoveContainer" containerID="33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41" Nov 03 21:00:05.202071 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.202047 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64_odh-ai-gateway-infra(210ea6fe-9453-40e0-8fbe-22eef3f023a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" Nov 03 21:00:05.203575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.203555 5932 generic.go:356] "Generic (PLEG): container finished" podID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerID="c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd" exitCode=60 Nov 03 21:00:05.203708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.203670 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerDied","Data":"c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd"} Nov 03 21:00:05.203887 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.203871 5932 scope.go:122] "RemoveContainer" containerID="c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd" Nov 03 21:00:05.204126 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.204106 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms_odh-ai-gateway-infra(b61bafdc-5ebf-4043-8e2d-920f48539295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" Nov 03 21:00:05.205599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.205578 5932 generic.go:356] "Generic (PLEG): container finished" podID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerID="3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef" exitCode=60 Nov 03 21:00:05.205719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.205647 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerDied","Data":"3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef"} Nov 03 21:00:05.205880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.205865 5932 scope.go:122] "RemoveContainer" containerID="3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef" Nov 03 21:00:05.206138 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.206118 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq_odh-ai-gateway-infra(508ccfc2-5101-4881-9340-7d4f701c1b01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" Nov 03 21:00:05.207692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.207671 5932 generic.go:356] "Generic (PLEG): container finished" podID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerID="c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9" exitCode=60 Nov 03 21:00:05.207790 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.207776 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerDied","Data":"c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9"} Nov 03 21:00:05.207990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.207972 5932 scope.go:122] "RemoveContainer" containerID="c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9" Nov 03 21:00:05.208240 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.208215 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd_odh-ai-gateway-infra(f2a85c24-21c3-4252-88e4-96b92aeddb94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" Nov 03 21:00:05.209367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.209359 5932 scope.go:122] "RemoveContainer" containerID="a850c2d2f78fac4ee58ae6019b566dde09a3ddc740aea93bca6b940d2296d167" Nov 03 21:00:05.209798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.209775 5932 generic.go:356] "Generic (PLEG): container finished" podID="1decd89c-8bce-4bd9-b698-68910d386e95" containerID="997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c" exitCode=60 Nov 03 21:00:05.209798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.209793 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerDied","Data":"997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c"} Nov 03 21:00:05.210039 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.210021 5932 scope.go:122] "RemoveContainer" containerID="997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c" Nov 03 21:00:05.210312 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:05.210288 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd_odh-ai-gateway-infra(1decd89c-8bce-4bd9-b698-68910d386e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" Nov 03 21:00:05.223598 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.223579 5932 scope.go:122] "RemoveContainer" containerID="7f3694a8cc5e96e92561ccba6e75f6da6a48708dccdedc82a0cdc0ec7502c910" Nov 03 21:00:05.239814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.239436 5932 scope.go:122] "RemoveContainer" containerID="019671d5abcda3e528994128a986ffd9058511dd01ecae2063d50ff8e31badae" Nov 03 21:00:05.256110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.256085 5932 scope.go:122] "RemoveContainer" containerID="26422778ed9fc45b274f0fc79018ec4985966726e95ba1f4536c53059542b736" Nov 03 21:00:05.270652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.270611 5932 scope.go:122] "RemoveContainer" containerID="0831848b1758fd4f14e5953f3707628233b3ecbbae5dbcc271d3a639465dd101" Nov 03 21:00:05.285218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.285198 5932 scope.go:122] "RemoveContainer" containerID="e33bbfdd776d47d229e94ff00b5a1ba1e7941fe88c14ecbd0818fdd86ff4b7be" Nov 03 21:00:05.300969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.300942 5932 scope.go:122] "RemoveContainer" containerID="a40e208538dfe28650c6f95eff5b35cdf021031c4a1b459bf8b6aeb3bb321806" Nov 03 21:00:05.315699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:05.315675 5932 scope.go:122] "RemoveContainer" containerID="f55944d18c8da1a0e302aced866e37af1b9b284e38cf0343decd453164e5ca6b" Nov 03 21:00:06.115871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.115839 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz"] Nov 03 21:00:06.122246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.122223 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.226299 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.226222 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-37c5b0-openshift-default-dockercfg-m8jmf\"" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258187 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258241 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258273 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258420 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258459 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gsbn\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258553 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258587 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258748 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.258833 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.284686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.262320 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz"] Nov 03 21:00:06.378974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.378922 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.379263 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.379227 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.379532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.379504 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.379694 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.379668 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gsbn\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.379988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.379968 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.380148 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.380130 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.380470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.380451 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.380826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.380709 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.381156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.381137 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.386101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.385874 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.387989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.385955 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.388143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.386677 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.388296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.386681 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.388413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.387846 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.476678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.466868 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.490775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.488334 5932 scope.go:122] "RemoveContainer" containerID="997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c" Nov 03 21:00:06.576074 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.574189 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.582691 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.581692 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd_odh-ai-gateway-infra(1decd89c-8bce-4bd9-b698-68910d386e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" Nov 03 21:00:06.582691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.582275 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.655967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.653390 5932 scope.go:122] "RemoveContainer" containerID="a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda" Nov 03 21:00:06.669067 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.669021 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r_odh-ai-gateway-infra(5ba61f92-9dde-40ea-baa8-954bcb70c415)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" Nov 03 21:00:06.672518 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.671859 5932 scope.go:122] "RemoveContainer" containerID="1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec" Nov 03 21:00:06.674111 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.674007 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz_odh-ai-gateway-infra(6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" Nov 03 21:00:06.677189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.677163 5932 scope.go:122] "RemoveContainer" containerID="4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf" Nov 03 21:00:06.679014 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.678975 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh_odh-ai-gateway-infra(53138c21-35f3-44a8-a8e3-31000aa44218)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" Nov 03 21:00:06.683581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.683511 5932 scope.go:122] "RemoveContainer" containerID="95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec" Nov 03 21:00:06.684966 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.684938 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz_odh-ai-gateway-infra(d7a0522c-60ab-4663-9f9a-bf8179386990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" Nov 03 21:00:06.692754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.689983 5932 scope.go:122] "RemoveContainer" containerID="33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41" Nov 03 21:00:06.692754 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.690519 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64_odh-ai-gateway-infra(210ea6fe-9453-40e0-8fbe-22eef3f023a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" Nov 03 21:00:06.698471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.698448 5932 scope.go:122] "RemoveContainer" containerID="c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd" Nov 03 21:00:06.699142 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.699112 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms_odh-ai-gateway-infra(b61bafdc-5ebf-4043-8e2d-920f48539295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" Nov 03 21:00:06.701521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.701506 5932 scope.go:122] "RemoveContainer" containerID="3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef" Nov 03 21:00:06.702652 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.702592 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq_odh-ai-gateway-infra(508ccfc2-5101-4881-9340-7d4f701c1b01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" Nov 03 21:00:06.703383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.703309 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gsbn\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn\") pod \"e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:06.708248 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.706038 5932 scope.go:122] "RemoveContainer" containerID="c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9" Nov 03 21:00:06.708248 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:06.706334 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd_odh-ai-gateway-infra(f2a85c24-21c3-4252-88e4-96b92aeddb94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" Nov 03 21:00:06.894645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:06.873426 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:07.018481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.018445 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs"] Nov 03 21:00:07.026231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.026205 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.075233 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.075192 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.075538 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.075516 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.075791 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.075771 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.075957 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.075940 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.076073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.076056 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kll96\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.076205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.076190 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.076398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.076381 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.076497 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.076482 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.076660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.076645 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.085752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.085366 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-af2e8a-openshift-default-dockercfg-kds9t\"" Nov 03 21:00:07.088043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.088017 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs"] Nov 03 21:00:07.215802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.215702 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.215802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.215788 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.216042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.215824 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kll96\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.216042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.215873 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.216155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.216058 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.216155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.216085 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.216273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.216162 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.216273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.216208 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.235457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.234366 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.237245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.237217 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.237765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.237740 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.239022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.238151 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.239022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.238968 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.239543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.239476 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.269937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.268656 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.281455 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.270940 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.296537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.296502 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kll96\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.296786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.296768 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token\") pod \"e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.345272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.345242 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:07.447986 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.447952 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz"] Nov 03 21:00:07.448389 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:07.448359 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d57f71_663f_4a44_a0c7_59db19c7ad1e.slice/crio-d96abf8feb0f794db9e2bdc2ae19291cbe8406eb304c055e0ba3f0d85a73c13e WatchSource:0}: Error finding container d96abf8feb0f794db9e2bdc2ae19291cbe8406eb304c055e0ba3f0d85a73c13e: Status 404 returned error can't find the container with id d96abf8feb0f794db9e2bdc2ae19291cbe8406eb304c055e0ba3f0d85a73c13e Nov 03 21:00:07.451912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.451880 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:07.452016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.451963 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:07.452016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.452003 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:07.716126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.716092 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" event={"ID":"80d57f71-663f-4a44-a0c7-59db19c7ad1e","Type":"ContainerStarted","Data":"0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47"} Nov 03 21:00:07.716334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.716149 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" event={"ID":"80d57f71-663f-4a44-a0c7-59db19c7ad1e","Type":"ContainerStarted","Data":"d96abf8feb0f794db9e2bdc2ae19291cbe8406eb304c055e0ba3f0d85a73c13e"} Nov 03 21:00:07.762244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.762152 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs"] Nov 03 21:00:07.766607 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:07.766579 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda847dd75_4df4_45dd_8511_7bde98fd7c92.slice/crio-8e59fc2e5231e147952e961560122fd810836455658b7b8244cb41a70ebe10a6 WatchSource:0}: Error finding container 8e59fc2e5231e147952e961560122fd810836455658b7b8244cb41a70ebe10a6: Status 404 returned error can't find the container with id 8e59fc2e5231e147952e961560122fd810836455658b7b8244cb41a70ebe10a6 Nov 03 21:00:07.769166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.769127 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:07.769271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.769197 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:07.769271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.769226 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:07.894647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.891753 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:07.894647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:07.894083 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podStartSLOduration=1.894070636 podStartE2EDuration="1.894070636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:07.888472961 +0000 UTC m=+1815.953714436" watchObservedRunningTime="2026-11-03 21:00:07.894070636 +0000 UTC m=+1815.959312108" Nov 03 21:00:08.735729 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:08.735681 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" event={"ID":"a847dd75-4df4-45dd-8511-7bde98fd7c92","Type":"ContainerStarted","Data":"6792df5727ef0b3ef240dfcc45b089ce09a4b2e18edd1c066a212bef477ae7c9"} Nov 03 21:00:08.736215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:08.735792 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" event={"ID":"a847dd75-4df4-45dd-8511-7bde98fd7c92","Type":"ContainerStarted","Data":"8e59fc2e5231e147952e961560122fd810836455658b7b8244cb41a70ebe10a6"} Nov 03 21:00:08.858195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:08.858143 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podStartSLOduration=2.858129076 podStartE2EDuration="2.858129076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:08.854092123 +0000 UTC m=+1816.919333596" watchObservedRunningTime="2026-11-03 21:00:08.858129076 +0000 UTC m=+1816.923370616" Nov 03 21:00:08.898918 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:08.898888 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:09.345883 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:09.345844 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:09.352237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:09.352207 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:09.744740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:09.744702 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:09.745227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:09.744762 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:09.745699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:09.745677 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:00:09.745816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:09.745744 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:00:14.153917 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:14.153865 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:00:15.044654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:15.044595 5932 scope.go:122] "RemoveContainer" containerID="4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf" Nov 03 21:00:15.840346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:15.840249 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerStarted","Data":"513db84366a5837e7523b750f65b556b4c7555965e7e436419597e995f423814"} Nov 03 21:00:16.850943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:16.850909 5932 generic.go:356] "Generic (PLEG): container finished" podID="53138c21-35f3-44a8-a8e3-31000aa44218" containerID="513db84366a5837e7523b750f65b556b4c7555965e7e436419597e995f423814" exitCode=60 Nov 03 21:00:16.851357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:16.850982 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerDied","Data":"513db84366a5837e7523b750f65b556b4c7555965e7e436419597e995f423814"} Nov 03 21:00:16.851357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:16.851046 5932 scope.go:122] "RemoveContainer" containerID="4143f5367b97f14e2f85a6f1ac0b45ea88b7cc15f2eb3f695f9c934b6be17eaf" Nov 03 21:00:16.851482 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:16.851465 5932 scope.go:122] "RemoveContainer" containerID="513db84366a5837e7523b750f65b556b4c7555965e7e436419597e995f423814" Nov 03 21:00:16.855398 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:16.852605 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh_odh-ai-gateway-infra(53138c21-35f3-44a8-a8e3-31000aa44218)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" Nov 03 21:00:16.976658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:16.976604 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh"] Nov 03 21:00:17.044778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.044745 5932 scope.go:122] "RemoveContainer" containerID="c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd" Nov 03 21:00:17.044958 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.044931 5932 scope.go:122] "RemoveContainer" containerID="a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda" Nov 03 21:00:17.045108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.045086 5932 scope.go:122] "RemoveContainer" containerID="3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef" Nov 03 21:00:17.045257 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.045233 5932 scope.go:122] "RemoveContainer" containerID="c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9" Nov 03 21:00:17.045321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.045309 5932 scope.go:122] "RemoveContainer" containerID="997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c" Nov 03 21:00:17.046035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.045486 5932 scope.go:122] "RemoveContainer" containerID="95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec" Nov 03 21:00:17.861556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.861519 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerStarted","Data":"ad42b664603c59f55486169b482bd95145424c6a0f1620102cd5077c1920bb24"} Nov 03 21:00:17.863890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.863860 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerStarted","Data":"8a497a7e7b7bf6954f69bf554d50d9a89daed1a1d669f4acd606a40b70e8fe78"} Nov 03 21:00:17.865905 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.865882 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerStarted","Data":"5587b96c9896f0d6b611a358d700bc8be4b498f2d2bcfaed5ae15c32985d5805"} Nov 03 21:00:17.869542 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.869514 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerStarted","Data":"1b50bf0720ac832357d2878bf364d26e5e3aaafa6804873009dcb1e32592f5d0"} Nov 03 21:00:17.871774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.871752 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerStarted","Data":"1dfe7d6085bdc20f62af0ac478b85f0bf2998db7c01336bbfc915491ebc26e50"} Nov 03 21:00:17.873742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:17.873722 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerStarted","Data":"94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13"} Nov 03 21:00:18.044817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.044570 5932 scope.go:122] "RemoveContainer" containerID="1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec" Nov 03 21:00:18.045524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.044843 5932 scope.go:122] "RemoveContainer" containerID="33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41" Nov 03 21:00:18.049078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.049055 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:18.161905 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.161878 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff\" (UniqueName: \"kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff\") pod \"53138c21-35f3-44a8-a8e3-31000aa44218\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " Nov 03 21:00:18.162077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.161958 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca\") pod \"53138c21-35f3-44a8-a8e3-31000aa44218\" (UID: \"53138c21-35f3-44a8-a8e3-31000aa44218\") " Nov 03 21:00:18.162269 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.162246 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca" pod "53138c21-35f3-44a8-a8e3-31000aa44218" (UID: "53138c21-35f3-44a8-a8e3-31000aa44218"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:18.162926 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.162903 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53138c21-35f3-44a8-a8e3-31000aa44218-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.227207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.227173 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff" pod "53138c21-35f3-44a8-a8e3-31000aa44218" (UID: "53138c21-35f3-44a8-a8e3-31000aa44218"). InnerVolumeSpecName "kube-api-access-d6zff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:18.264596 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.264525 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6zff\" (UniqueName: \"kubernetes.io/projected/53138c21-35f3-44a8-a8e3-31000aa44218-kube-api-access-d6zff\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:18.551989 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.551953 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ccfc2_5101_4881_9340_7d4f701c1b01.slice/crio-94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ccfc2_5101_4881_9340_7d4f701c1b01.slice/crio-conmon-94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a85c24_21c3_4252_88e4_96b92aeddb94.slice/crio-conmon-ad42b664603c59f55486169b482bd95145424c6a0f1620102cd5077c1920bb24.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:00:18.884097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.884060 5932 generic.go:356] "Generic (PLEG): container finished" podID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerID="1dfe7d6085bdc20f62af0ac478b85f0bf2998db7c01336bbfc915491ebc26e50" exitCode=60 Nov 03 21:00:18.884565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.884130 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerDied","Data":"1dfe7d6085bdc20f62af0ac478b85f0bf2998db7c01336bbfc915491ebc26e50"} Nov 03 21:00:18.884565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.884208 5932 scope.go:122] "RemoveContainer" containerID="c56b11e78b3dfd1cad911f8cb714218a7ce7e6c3b585d36d21d78d23299cebfd" Nov 03 21:00:18.884724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.884703 5932 scope.go:122] "RemoveContainer" containerID="1dfe7d6085bdc20f62af0ac478b85f0bf2998db7c01336bbfc915491ebc26e50" Nov 03 21:00:18.885227 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.885196 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms_odh-ai-gateway-infra(b61bafdc-5ebf-4043-8e2d-920f48539295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" Nov 03 21:00:18.887570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.887549 5932 generic.go:356] "Generic (PLEG): container finished" podID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerID="94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13" exitCode=60 Nov 03 21:00:18.887723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.887630 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerDied","Data":"94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13"} Nov 03 21:00:18.888070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.888051 5932 scope.go:122] "RemoveContainer" containerID="94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13" Nov 03 21:00:18.888447 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.888421 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq_odh-ai-gateway-infra(508ccfc2-5101-4881-9340-7d4f701c1b01)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" Nov 03 21:00:18.889814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.889786 5932 generic.go:356] "Generic (PLEG): container finished" podID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerID="ad42b664603c59f55486169b482bd95145424c6a0f1620102cd5077c1920bb24" exitCode=60 Nov 03 21:00:18.889913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.889817 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerDied","Data":"ad42b664603c59f55486169b482bd95145424c6a0f1620102cd5077c1920bb24"} Nov 03 21:00:18.890211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.890196 5932 scope.go:122] "RemoveContainer" containerID="ad42b664603c59f55486169b482bd95145424c6a0f1620102cd5077c1920bb24" Nov 03 21:00:18.890550 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.890523 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd_odh-ai-gateway-infra(f2a85c24-21c3-4252-88e4-96b92aeddb94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" Nov 03 21:00:18.892118 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.892100 5932 generic.go:356] "Generic (PLEG): container finished" podID="1decd89c-8bce-4bd9-b698-68910d386e95" containerID="8a497a7e7b7bf6954f69bf554d50d9a89daed1a1d669f4acd606a40b70e8fe78" exitCode=60 Nov 03 21:00:18.892221 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.892176 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerDied","Data":"8a497a7e7b7bf6954f69bf554d50d9a89daed1a1d669f4acd606a40b70e8fe78"} Nov 03 21:00:18.892552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.892535 5932 scope.go:122] "RemoveContainer" containerID="8a497a7e7b7bf6954f69bf554d50d9a89daed1a1d669f4acd606a40b70e8fe78" Nov 03 21:00:18.892925 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.892901 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd_odh-ai-gateway-infra(1decd89c-8bce-4bd9-b698-68910d386e95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" Nov 03 21:00:18.895287 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.895265 5932 generic.go:356] "Generic (PLEG): container finished" podID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerID="5587b96c9896f0d6b611a358d700bc8be4b498f2d2bcfaed5ae15c32985d5805" exitCode=60 Nov 03 21:00:18.895371 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.895326 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerDied","Data":"5587b96c9896f0d6b611a358d700bc8be4b498f2d2bcfaed5ae15c32985d5805"} Nov 03 21:00:18.895766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.895746 5932 scope.go:122] "RemoveContainer" containerID="5587b96c9896f0d6b611a358d700bc8be4b498f2d2bcfaed5ae15c32985d5805" Nov 03 21:00:18.896060 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.896033 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r_odh-ai-gateway-infra(5ba61f92-9dde-40ea-baa8-954bcb70c415)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" Nov 03 21:00:18.897374 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.897352 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerStarted","Data":"5db360f8281748522487e28477dc1fda181da0a007b6c9a68c88d1563014c642"} Nov 03 21:00:18.898833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.898815 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" Nov 03 21:00:18.898833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.898826 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh" event={"ID":"53138c21-35f3-44a8-a8e3-31000aa44218","Type":"ContainerDied","Data":"2fade6a785696f407f216a659ea9d62be2f15eccf6d1e03cd50f53fab1da6e2d"} Nov 03 21:00:18.901773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.901754 5932 scope.go:122] "RemoveContainer" containerID="3d727c34d3c09dfff19ffcbefbf78a41b7d47a1e646b140fe3ad5823a1e7edef" Nov 03 21:00:18.902175 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.902154 5932 generic.go:356] "Generic (PLEG): container finished" podID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerID="1b50bf0720ac832357d2878bf364d26e5e3aaafa6804873009dcb1e32592f5d0" exitCode=60 Nov 03 21:00:18.902249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.902219 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerDied","Data":"1b50bf0720ac832357d2878bf364d26e5e3aaafa6804873009dcb1e32592f5d0"} Nov 03 21:00:18.902588 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.902560 5932 scope.go:122] "RemoveContainer" containerID="1b50bf0720ac832357d2878bf364d26e5e3aaafa6804873009dcb1e32592f5d0" Nov 03 21:00:18.903058 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:18.903035 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz_odh-ai-gateway-infra(d7a0522c-60ab-4663-9f9a-bf8179386990)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" Nov 03 21:00:18.904335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.904315 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerStarted","Data":"991418e49694bee78c6fd66f719d5189c41562015359c329aa05730de0cbb90b"} Nov 03 21:00:18.917324 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.917307 5932 scope.go:122] "RemoveContainer" containerID="c98ac21851b6709b1030a064f7e5b65175ed542a1d6f59cdb522eba3ba0c95d9" Nov 03 21:00:18.931880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.931856 5932 scope.go:122] "RemoveContainer" containerID="997542e433fe47ac271494a1f06af3f4441df966712514e1a6c19044a13ab55c" Nov 03 21:00:18.946021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.946001 5932 scope.go:122] "RemoveContainer" containerID="a6b7050f0ba779132b9badbf3a38a2d0c4eb35b2789d8ef3df8c20c8b6737fda" Nov 03 21:00:18.960353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.960336 5932 scope.go:122] "RemoveContainer" containerID="513db84366a5837e7523b750f65b556b4c7555965e7e436419597e995f423814" Nov 03 21:00:18.969896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.969850 5932 scope.go:122] "RemoveContainer" containerID="95cb7d5db6c3f7eccd81442ee551b9adf8c8a9ffd10a420e1a5d18a10c331dec" Nov 03 21:00:18.980919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:18.980896 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd"] Nov 03 21:00:19.096285 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.096258 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz"] Nov 03 21:00:19.204925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.204893 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd"] Nov 03 21:00:19.312520 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.312487 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r"] Nov 03 21:00:19.455398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.455094 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh"] Nov 03 21:00:19.459659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.459618 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-2e3ea2-29895660-8npdh"] Nov 03 21:00:19.552136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.552107 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms"] Nov 03 21:00:19.654318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.654283 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq"] Nov 03 21:00:19.927220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.927190 5932 generic.go:356] "Generic (PLEG): container finished" podID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerID="5db360f8281748522487e28477dc1fda181da0a007b6c9a68c88d1563014c642" exitCode=60 Nov 03 21:00:19.927612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.927301 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerDied","Data":"5db360f8281748522487e28477dc1fda181da0a007b6c9a68c88d1563014c642"} Nov 03 21:00:19.927612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.927358 5932 scope.go:122] "RemoveContainer" containerID="1e9909a2c5e179f9a32d45e09bb6ae2a280510366671e72758dfc9d00b20ccec" Nov 03 21:00:19.927871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.927845 5932 scope.go:122] "RemoveContainer" containerID="5db360f8281748522487e28477dc1fda181da0a007b6c9a68c88d1563014c642" Nov 03 21:00:19.928338 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:19.928307 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz_odh-ai-gateway-infra(6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" Nov 03 21:00:19.951841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.949986 5932 generic.go:356] "Generic (PLEG): container finished" podID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerID="991418e49694bee78c6fd66f719d5189c41562015359c329aa05730de0cbb90b" exitCode=60 Nov 03 21:00:19.951841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.950101 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerDied","Data":"991418e49694bee78c6fd66f719d5189c41562015359c329aa05730de0cbb90b"} Nov 03 21:00:19.951841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:19.950522 5932 scope.go:122] "RemoveContainer" containerID="991418e49694bee78c6fd66f719d5189c41562015359c329aa05730de0cbb90b" Nov 03 21:00:19.952989 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:19.952954 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64_odh-ai-gateway-infra(210ea6fe-9453-40e0-8fbe-22eef3f023a5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" Nov 03 21:00:20.014856 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.012494 5932 scope.go:122] "RemoveContainer" containerID="33bfe6846305f9c9008c4debece4b57978904c4a6c5acbba5e0b08651eba6a41" Nov 03 21:00:20.062797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.062735 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" path="/var/lib/kubelet/pods/53138c21-35f3-44a8-a8e3-31000aa44218/volumes" Nov 03 21:00:20.175404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.175380 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:20.262059 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.254400 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca\") pod \"5ba61f92-9dde-40ea-baa8-954bcb70c415\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " Nov 03 21:00:20.262059 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.254467 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp\" (UniqueName: \"kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp\") pod \"5ba61f92-9dde-40ea-baa8-954bcb70c415\" (UID: \"5ba61f92-9dde-40ea-baa8-954bcb70c415\") " Nov 03 21:00:20.262059 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.256196 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca" pod "5ba61f92-9dde-40ea-baa8-954bcb70c415" (UID: "5ba61f92-9dde-40ea-baa8-954bcb70c415"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.267247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.265652 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp" pod "5ba61f92-9dde-40ea-baa8-954bcb70c415" (UID: "5ba61f92-9dde-40ea-baa8-954bcb70c415"). InnerVolumeSpecName "kube-api-access-jqfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.292864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.292754 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:20.326726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.326677 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:20.331739 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.331715 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:20.336366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.336341 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:20.344302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.344275 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:20.356354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.356329 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h\" (UniqueName: \"kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h\") pod \"b61bafdc-5ebf-4043-8e2d-920f48539295\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " Nov 03 21:00:20.356499 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.356383 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq\" (UniqueName: \"kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq\") pod \"f2a85c24-21c3-4252-88e4-96b92aeddb94\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " Nov 03 21:00:20.356684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.356635 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca\") pod \"f2a85c24-21c3-4252-88e4-96b92aeddb94\" (UID: \"f2a85c24-21c3-4252-88e4-96b92aeddb94\") " Nov 03 21:00:20.356796 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.356767 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca\") pod \"b61bafdc-5ebf-4043-8e2d-920f48539295\" (UID: \"b61bafdc-5ebf-4043-8e2d-920f48539295\") " Nov 03 21:00:20.357223 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.357082 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca" pod "f2a85c24-21c3-4252-88e4-96b92aeddb94" (UID: "f2a85c24-21c3-4252-88e4-96b92aeddb94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.357406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.357382 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2a85c24-21c3-4252-88e4-96b92aeddb94-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.357493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.357410 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ba61f92-9dde-40ea-baa8-954bcb70c415-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.357493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.357424 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqfsp\" (UniqueName: \"kubernetes.io/projected/5ba61f92-9dde-40ea-baa8-954bcb70c415-kube-api-access-jqfsp\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.357493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.357455 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca" pod "b61bafdc-5ebf-4043-8e2d-920f48539295" (UID: "b61bafdc-5ebf-4043-8e2d-920f48539295"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.359078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.359054 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h" pod "b61bafdc-5ebf-4043-8e2d-920f48539295" (UID: "b61bafdc-5ebf-4043-8e2d-920f48539295"). InnerVolumeSpecName "kube-api-access-p758h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.359240 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.359225 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq" pod "f2a85c24-21c3-4252-88e4-96b92aeddb94" (UID: "f2a85c24-21c3-4252-88e4-96b92aeddb94"). InnerVolumeSpecName "kube-api-access-sdvlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.464662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.464517 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s\" (UniqueName: \"kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s\") pod \"1decd89c-8bce-4bd9-b698-68910d386e95\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " Nov 03 21:00:20.464662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.464584 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca\") pod \"1decd89c-8bce-4bd9-b698-68910d386e95\" (UID: \"1decd89c-8bce-4bd9-b698-68910d386e95\") " Nov 03 21:00:20.464662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.464640 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca\") pod \"508ccfc2-5101-4881-9340-7d4f701c1b01\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " Nov 03 21:00:20.464959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.464728 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca\") pod \"d7a0522c-60ab-4663-9f9a-bf8179386990\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " Nov 03 21:00:20.464959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.464941 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx\" (UniqueName: \"kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx\") pod \"d7a0522c-60ab-4663-9f9a-bf8179386990\" (UID: \"d7a0522c-60ab-4663-9f9a-bf8179386990\") " Nov 03 21:00:20.465062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.465035 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm\" (UniqueName: \"kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm\") pod \"508ccfc2-5101-4881-9340-7d4f701c1b01\" (UID: \"508ccfc2-5101-4881-9340-7d4f701c1b01\") " Nov 03 21:00:20.466336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.465920 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b61bafdc-5ebf-4043-8e2d-920f48539295-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.466336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.465948 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p758h\" (UniqueName: \"kubernetes.io/projected/b61bafdc-5ebf-4043-8e2d-920f48539295-kube-api-access-p758h\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.466336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.465969 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdvlq\" (UniqueName: \"kubernetes.io/projected/f2a85c24-21c3-4252-88e4-96b92aeddb94-kube-api-access-sdvlq\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.469447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.469403 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca" pod "508ccfc2-5101-4881-9340-7d4f701c1b01" (UID: "508ccfc2-5101-4881-9340-7d4f701c1b01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.469447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.469430 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca" pod "d7a0522c-60ab-4663-9f9a-bf8179386990" (UID: "d7a0522c-60ab-4663-9f9a-bf8179386990"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.469729 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.469515 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca" pod "1decd89c-8bce-4bd9-b698-68910d386e95" (UID: "1decd89c-8bce-4bd9-b698-68910d386e95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:20.471599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.471569 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s" pod "1decd89c-8bce-4bd9-b698-68910d386e95" (UID: "1decd89c-8bce-4bd9-b698-68910d386e95"). InnerVolumeSpecName "kube-api-access-gjj5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.471714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.471610 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx" pod "d7a0522c-60ab-4663-9f9a-bf8179386990" (UID: "d7a0522c-60ab-4663-9f9a-bf8179386990"). InnerVolumeSpecName "kube-api-access-5q7kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.472488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.472452 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm" pod "508ccfc2-5101-4881-9340-7d4f701c1b01" (UID: "508ccfc2-5101-4881-9340-7d4f701c1b01"). InnerVolumeSpecName "kube-api-access-gz4jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:20.567730 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.567695 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ccfc2-5101-4881-9340-7d4f701c1b01-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.567730 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.567724 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a0522c-60ab-4663-9f9a-bf8179386990-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.567730 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.567734 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q7kx\" (UniqueName: \"kubernetes.io/projected/d7a0522c-60ab-4663-9f9a-bf8179386990-kube-api-access-5q7kx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.567965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.567743 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz4jm\" (UniqueName: \"kubernetes.io/projected/508ccfc2-5101-4881-9340-7d4f701c1b01-kube-api-access-gz4jm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.567965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.567752 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj5s\" (UniqueName: \"kubernetes.io/projected/1decd89c-8bce-4bd9-b698-68910d386e95-kube-api-access-gjj5s\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.567965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.567761 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1decd89c-8bce-4bd9-b698-68910d386e95-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:20.717842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.717753 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz"] Nov 03 21:00:20.867345 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.867312 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64"] Nov 03 21:00:20.964384 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.964354 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" event={"ID":"d7a0522c-60ab-4663-9f9a-bf8179386990","Type":"ContainerDied","Data":"0f90f105a8c890e908c0850e19a8f46666c927a5a64e45ff2ba0e95f4595695a"} Nov 03 21:00:20.964384 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.964376 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz" Nov 03 21:00:20.964831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.964413 5932 scope.go:122] "RemoveContainer" containerID="1b50bf0720ac832357d2878bf364d26e5e3aaafa6804873009dcb1e32592f5d0" Nov 03 21:00:20.972289 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.972198 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" event={"ID":"b61bafdc-5ebf-4043-8e2d-920f48539295","Type":"ContainerDied","Data":"4ce5f4d4a525e1a205db2cb45fad2d3cf206b40033db3e3a8c563ef0b71c71f7"} Nov 03 21:00:20.972445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.972341 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms" Nov 03 21:00:20.977226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.975049 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" event={"ID":"508ccfc2-5101-4881-9340-7d4f701c1b01","Type":"ContainerDied","Data":"53b7998aa4b4d7edb68fb69c8a6c3664b8df4797dd7b78ae144bc8359385ff25"} Nov 03 21:00:20.977226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.975076 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq" Nov 03 21:00:20.981106 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.980992 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" event={"ID":"f2a85c24-21c3-4252-88e4-96b92aeddb94","Type":"ContainerDied","Data":"583318c01e5a367697bb25f1bb61c607e4aaedd3159d8c28493e814a8de5afc3"} Nov 03 21:00:20.981192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.981116 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd" Nov 03 21:00:20.983807 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.983761 5932 scope.go:122] "RemoveContainer" containerID="1dfe7d6085bdc20f62af0ac478b85f0bf2998db7c01336bbfc915491ebc26e50" Nov 03 21:00:20.988329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.988229 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" Nov 03 21:00:20.988329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.988261 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd" event={"ID":"1decd89c-8bce-4bd9-b698-68910d386e95","Type":"ContainerDied","Data":"529e70e3c15ac33270cd3f82c8440d8ba73eb96505deb5ddc28289604695f684"} Nov 03 21:00:20.990476 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.990446 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" event={"ID":"5ba61f92-9dde-40ea-baa8-954bcb70c415","Type":"ContainerDied","Data":"d98fc86b5516cb3b8fa2538129341a3a9384292f7c25999842136cbeb2ab4f5b"} Nov 03 21:00:20.990659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:20.990555 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r" Nov 03 21:00:21.002488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.002464 5932 scope.go:122] "RemoveContainer" containerID="94f7e1a85735eaf060c6b9ea67dc4b3bbfbcc166e291e3f2958584d0afe9ba13" Nov 03 21:00:21.017171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.017153 5932 scope.go:122] "RemoveContainer" containerID="ad42b664603c59f55486169b482bd95145424c6a0f1620102cd5077c1920bb24" Nov 03 21:00:21.042924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.042824 5932 scope.go:122] "RemoveContainer" containerID="8a497a7e7b7bf6954f69bf554d50d9a89daed1a1d669f4acd606a40b70e8fe78" Nov 03 21:00:21.056004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.055758 5932 scope.go:122] "RemoveContainer" containerID="5587b96c9896f0d6b611a358d700bc8be4b498f2d2bcfaed5ae15c32985d5805" Nov 03 21:00:21.316406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.316379 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:21.318451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.318434 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:21.405469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.405436 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8\" (UniqueName: \"kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8\") pod \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " Nov 03 21:00:21.405664 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.405602 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca\") pod \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " Nov 03 21:00:21.405779 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.405761 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl\" (UniqueName: \"kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl\") pod \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\" (UID: \"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9\") " Nov 03 21:00:21.405843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.405826 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca\") pod \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\" (UID: \"210ea6fe-9453-40e0-8fbe-22eef3f023a5\") " Nov 03 21:00:21.406021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.405993 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca" pod "6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" (UID: "6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.406231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.406203 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca" pod "210ea6fe-9453-40e0-8fbe-22eef3f023a5" (UID: "210ea6fe-9453-40e0-8fbe-22eef3f023a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:00:21.406690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.406663 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.406802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.406694 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/210ea6fe-9453-40e0-8fbe-22eef3f023a5-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.410300 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.408484 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8" pod "210ea6fe-9453-40e0-8fbe-22eef3f023a5" (UID: "210ea6fe-9453-40e0-8fbe-22eef3f023a5"). InnerVolumeSpecName "kube-api-access-h86d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.410300 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.408598 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl" pod "6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" (UID: "6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9"). InnerVolumeSpecName "kube-api-access-chtsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:00:21.507835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.507752 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86d8\" (UniqueName: \"kubernetes.io/projected/210ea6fe-9453-40e0-8fbe-22eef3f023a5-kube-api-access-h86d8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.507835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.507782 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chtsl\" (UniqueName: \"kubernetes.io/projected/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9-kube-api-access-chtsl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:00:21.734998 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.734966 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms"] Nov 03 21:00:21.740471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.740445 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e68f8112-29895660-bdpms"] Nov 03 21:00:21.837613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.837524 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd"] Nov 03 21:00:21.841679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:21.841655 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895660-qpjvd"] Nov 03 21:00:22.005574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.005545 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" Nov 03 21:00:22.005574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.005567 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz" event={"ID":"6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9","Type":"ContainerDied","Data":"49cd16d3555cff4d00fcf7e0011ddac3ce10afa57e4760c2963d5151935d1ba7"} Nov 03 21:00:22.006102 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.005643 5932 scope.go:122] "RemoveContainer" containerID="5db360f8281748522487e28477dc1fda181da0a007b6c9a68c88d1563014c642" Nov 03 21:00:22.008155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.008134 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" event={"ID":"210ea6fe-9453-40e0-8fbe-22eef3f023a5","Type":"ContainerDied","Data":"beaca0764e24ca4368cd74478571abc4ec1a3706dc959ee458b902ef9799c60e"} Nov 03 21:00:22.008155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.008144 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64" Nov 03 21:00:22.018219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.018199 5932 scope.go:122] "RemoveContainer" containerID="991418e49694bee78c6fd66f719d5189c41562015359c329aa05730de0cbb90b" Nov 03 21:00:22.055577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.055542 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" path="/var/lib/kubelet/pods/b61bafdc-5ebf-4043-8e2d-920f48539295/volumes" Nov 03 21:00:22.056477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.056446 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" path="/var/lib/kubelet/pods/f2a85c24-21c3-4252-88e4-96b92aeddb94/volumes" Nov 03 21:00:22.165484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.165450 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq"] Nov 03 21:00:22.173038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.173001 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ba7e83-29895660-ksgsq"] Nov 03 21:00:22.276169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.276129 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz"] Nov 03 21:00:22.287860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.287827 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895660-4tvfz"] Nov 03 21:00:22.390895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.390862 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd"] Nov 03 21:00:22.400525 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.400493 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-1e6fe1-29895660-srkjd"] Nov 03 21:00:22.634645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.630887 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r"] Nov 03 21:00:22.635594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.635555 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-5b12d6-29895660-m826r"] Nov 03 21:00:22.778911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.767679 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz"] Nov 03 21:00:22.796651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.788403 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-cc5d0f-29895660-bvrfz"] Nov 03 21:00:22.897178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.895680 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64"] Nov 03 21:00:22.902216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.902177 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-2a5ec9-29895660-txp64"] Nov 03 21:00:22.907486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.907456 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p"] Nov 03 21:00:22.908464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908436 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:22.908464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908460 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908472 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908479 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908490 5932 cpu_manager.go:414] "containerMap: removing container" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908497 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908517 5932 cpu_manager.go:414] "containerMap: removing container" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908524 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908560 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908567 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908580 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908587 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908613 5932 cpu_manager.go:414] "containerMap: removing container" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908634 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908649 5932 cpu_manager.go:414] "containerMap: removing container" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908657 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908667 5932 cpu_manager.go:414] "containerMap: removing container" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.908671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908675 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908690 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908697 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908711 5932 cpu_manager.go:414] "containerMap: removing container" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908719 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908730 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908737 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908755 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908763 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908801 5932 cpu_manager.go:414] "containerMap: removing container" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908808 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908823 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908830 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908867 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908874 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908884 5932 cpu_manager.go:414] "containerMap: removing container" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908891 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908900 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908906 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908920 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908927 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908937 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.908944 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909131 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909148 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909158 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909168 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909176 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909184 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909193 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909205 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909214 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909224 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909235 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909245 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909255 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909264 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909272 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909280 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909289 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909297 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909309 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909320 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:22.909477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.909334 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:22.915563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.915537 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:22.948259 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.948231 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p"] Nov 03 21:00:22.968879 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.968850 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-37c5b0-dockercfg-sfw9v\"" Nov 03 21:00:22.969030 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:22.968848 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-37c5b0\"" Nov 03 21:00:23.100698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.052915 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvlvx\" (UniqueName: \"kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx\") pod \"payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.100698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.053011 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.133841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.133806 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4"] Nov 03 21:00:23.136176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136145 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:23.136444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136428 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:23.136548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136536 5932 cpu_manager.go:414] "containerMap: removing container" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:23.136661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136644 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:23.136824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136772 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:23.136915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136905 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:23.137004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.136996 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:23.137065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137055 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:23.137147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137137 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:23.137216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137208 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:23.137284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137276 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:23.137350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137342 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61bafdc-5ebf-4043-8e2d-920f48539295" containerName="cleanup" Nov 03 21:00:23.137421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137413 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:23.137498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137489 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" containerName="cleanup" Nov 03 21:00:23.137800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137787 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" containerName="cleanup" Nov 03 21:00:23.137897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137889 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" containerName="cleanup" Nov 03 21:00:23.137969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.137961 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" containerName="cleanup" Nov 03 21:00:23.138079 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.138031 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a85c24-21c3-4252-88e4-96b92aeddb94" containerName="cleanup" Nov 03 21:00:23.138156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.138149 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="53138c21-35f3-44a8-a8e3-31000aa44218" containerName="cleanup" Nov 03 21:00:23.138220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.138212 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" containerName="cleanup" Nov 03 21:00:23.151291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.151217 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4"] Nov 03 21:00:23.151587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.151571 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.154917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.154887 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvlvx\" (UniqueName: \"kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx\") pod \"payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.155833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.155810 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.157206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.156723 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.242007 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.240058 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvlvx\" (UniqueName: \"kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx\") pod \"payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.270466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.259806 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.270466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.259865 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84h6x\" (UniqueName: \"kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x\") pod \"payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.362055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.362021 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.362055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.362061 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84h6x\" (UniqueName: \"kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x\") pod \"payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.362757 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.362735 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.419366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.419285 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84h6x\" (UniqueName: \"kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x\") pod \"payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.481887 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.481855 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:23.531635 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.531590 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:23.871673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.871541 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4"] Nov 03 21:00:23.918405 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:23.918378 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb171c86_c01b_4785_ad64_da20394c9879.slice/crio-811602b29d2be5a03a905444bfe7ae5310ddeab4db0b917125bfdc705aa0d849 WatchSource:0}: Error finding container 811602b29d2be5a03a905444bfe7ae5310ddeab4db0b917125bfdc705aa0d849: Status 404 returned error can't find the container with id 811602b29d2be5a03a905444bfe7ae5310ddeab4db0b917125bfdc705aa0d849 Nov 03 21:00:23.927455 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:23.927429 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p"] Nov 03 21:00:23.933903 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:23.933872 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af792c6_f691_489a_a846_060155aa0ce0.slice/crio-6be6fe109d89e48bf13e1bb2255256043818ccb8f8e2a28a5e08571e844af701 WatchSource:0}: Error finding container 6be6fe109d89e48bf13e1bb2255256043818ccb8f8e2a28a5e08571e844af701: Status 404 returned error can't find the container with id 6be6fe109d89e48bf13e1bb2255256043818ccb8f8e2a28a5e08571e844af701 Nov 03 21:00:24.055033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.055008 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1decd89c-8bce-4bd9-b698-68910d386e95" path="/var/lib/kubelet/pods/1decd89c-8bce-4bd9-b698-68910d386e95/volumes" Nov 03 21:00:24.055466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.055452 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210ea6fe-9453-40e0-8fbe-22eef3f023a5" path="/var/lib/kubelet/pods/210ea6fe-9453-40e0-8fbe-22eef3f023a5/volumes" Nov 03 21:00:24.056759 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.056730 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508ccfc2-5101-4881-9340-7d4f701c1b01" path="/var/lib/kubelet/pods/508ccfc2-5101-4881-9340-7d4f701c1b01/volumes" Nov 03 21:00:24.057547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.057524 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba61f92-9dde-40ea-baa8-954bcb70c415" path="/var/lib/kubelet/pods/5ba61f92-9dde-40ea-baa8-954bcb70c415/volumes" Nov 03 21:00:24.058338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.058314 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9" path="/var/lib/kubelet/pods/6dc2f932-cec9-4504-8b5e-cd6d9a3a62d9/volumes" Nov 03 21:00:24.059543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.059518 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a0522c-60ab-4663-9f9a-bf8179386990" path="/var/lib/kubelet/pods/d7a0522c-60ab-4663-9f9a-bf8179386990/volumes" Nov 03 21:00:24.133222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.133191 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" event={"ID":"5af792c6-f691-489a-a846-060155aa0ce0","Type":"ContainerStarted","Data":"15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9"} Nov 03 21:00:24.133668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.133244 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" event={"ID":"5af792c6-f691-489a-a846-060155aa0ce0","Type":"ContainerStarted","Data":"6be6fe109d89e48bf13e1bb2255256043818ccb8f8e2a28a5e08571e844af701"} Nov 03 21:00:24.133668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.133391 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:24.135095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.135071 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" event={"ID":"bb171c86-c01b-4785-ad64-da20394c9879","Type":"ContainerStarted","Data":"ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa"} Nov 03 21:00:24.135197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.135107 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" event={"ID":"bb171c86-c01b-4785-ad64-da20394c9879","Type":"ContainerStarted","Data":"811602b29d2be5a03a905444bfe7ae5310ddeab4db0b917125bfdc705aa0d849"} Nov 03 21:00:24.135197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.135126 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:24.246157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.246107 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" podStartSLOduration=2.246093998 podStartE2EDuration="2.246093998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:24.242063715 +0000 UTC m=+1832.307305201" watchObservedRunningTime="2026-11-03 21:00:24.246093998 +0000 UTC m=+1832.311335481" Nov 03 21:00:24.354272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:24.354219 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" podStartSLOduration=1.354202666 podStartE2EDuration="1.354202666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:24.351059631 +0000 UTC m=+1832.416301115" watchObservedRunningTime="2026-11-03 21:00:24.354202666 +0000 UTC m=+1832.419444138" Nov 03 21:00:30.694246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.688906 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz"] Nov 03 21:00:30.733877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.733840 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz"] Nov 03 21:00:30.734243 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.734218 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:30.817646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.765454 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:30.830658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.820888 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mgf\" (UniqueName: \"kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf\") pod \"payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:30.830658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.808052 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-af2e8a-dockercfg-wrtmn\"" Nov 03 21:00:30.880545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.880509 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-af2e8a\"" Nov 03 21:00:30.935903 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.935857 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:30.955700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.948469 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mgf\" (UniqueName: \"kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf\") pod \"payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:31.010080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:30.937087 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume\") pod \"payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:31.033675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.010343 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn"] Nov 03 21:00:31.086076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.085990 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mgf\" (UniqueName: \"kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf\") pod \"payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:31.095155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.095123 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn"] Nov 03 21:00:31.095323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.095245 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.161714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.161668 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:31.211870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.211780 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.212049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.212001 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfb2v\" (UniqueName: \"kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v\") pod \"payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.313982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.313842 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfb2v\" (UniqueName: \"kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v\") pod \"payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.314397 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.314003 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.314793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.314748 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.375708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.375639 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfb2v\" (UniqueName: \"kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v\") pod \"payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.445644 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.444980 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:31.581679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.579528 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz"] Nov 03 21:00:31.584030 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:31.584003 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c572b5_3c9b_4272_ac29_fbb79385cc85.slice/crio-9be14dc3c7d115333980474770b1a1116b184d59b6fcb2421c871982e14a969e WatchSource:0}: Error finding container 9be14dc3c7d115333980474770b1a1116b184d59b6fcb2421c871982e14a969e: Status 404 returned error can't find the container with id 9be14dc3c7d115333980474770b1a1116b184d59b6fcb2421c871982e14a969e Nov 03 21:00:31.894956 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:31.894923 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn"] Nov 03 21:00:32.571247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.571207 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" event={"ID":"b1c572b5-3c9b-4272-ac29-fbb79385cc85","Type":"ContainerStarted","Data":"f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045"} Nov 03 21:00:32.571414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.571278 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:32.571414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.571317 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" event={"ID":"b1c572b5-3c9b-4272-ac29-fbb79385cc85","Type":"ContainerStarted","Data":"9be14dc3c7d115333980474770b1a1116b184d59b6fcb2421c871982e14a969e"} Nov 03 21:00:32.572977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.572939 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" event={"ID":"04832731-fd34-4c3a-aa4d-ea769ad586bd","Type":"ContainerStarted","Data":"316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8"} Nov 03 21:00:32.573117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.572998 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" event={"ID":"04832731-fd34-4c3a-aa4d-ea769ad586bd","Type":"ContainerStarted","Data":"db4a82406d59455d72db9892c84bbc3ce985f2ce2ecb1a9870b5bf78a1bbcbde"} Nov 03 21:00:32.573117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.573027 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:32.681661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.681592 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" podStartSLOduration=2.6815774980000002 podStartE2EDuration="2.681577498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:32.679418304 +0000 UTC m=+1840.744659777" watchObservedRunningTime="2026-11-03 21:00:32.681577498 +0000 UTC m=+1840.746818972" Nov 03 21:00:32.822145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:32.816687 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" podStartSLOduration=2.816664012 podStartE2EDuration="2.816664012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:32.800261009 +0000 UTC m=+1840.865502552" watchObservedRunningTime="2026-11-03 21:00:32.816664012 +0000 UTC m=+1840.881905486" Nov 03 21:00:35.048589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:35.048555 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk"] Nov 03 21:00:35.095764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:35.095729 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f518bcb-qt5tk"] Nov 03 21:00:35.148336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:35.148309 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:00:35.148510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:35.148376 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:00:36.056632 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:36.056578 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d2586e-63e1-4eb9-9df4-5a2ea1de15f2" path="/var/lib/kubelet/pods/19d2586e-63e1-4eb9-9df4-5a2ea1de15f2/volumes" Nov 03 21:00:37.620337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:37.620306 5932 generic.go:356] "Generic (PLEG): container finished" podID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerID="57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09" exitCode=0 Nov 03 21:00:37.620797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:37.620362 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" event={"ID":"5fcdb585-bf18-4dd5-a2dd-45a14c591520","Type":"ContainerDied","Data":"57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09"} Nov 03 21:00:37.620867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:37.620847 5932 scope.go:122] "RemoveContainer" containerID="57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09" Nov 03 21:00:37.622552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:37.622523 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:37.622616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:37.622602 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:37.622701 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:37.622649 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:38.643995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:38.643954 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" event={"ID":"5fcdb585-bf18-4dd5-a2dd-45a14c591520","Type":"ContainerStarted","Data":"4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e"} Nov 03 21:00:39.211943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:39.211904 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 21:00:39.213211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:39.213182 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:00:39.213360 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:39.213245 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 21:00:40.212696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:40.212660 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:00:40.213110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:40.212725 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 21:00:41.212481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:41.212444 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:00:41.212703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:41.212503 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 21:00:41.383271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:41.383239 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 21:00:42.212174 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:42.212137 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:00:42.212364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:42.212194 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 21:00:43.212307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:43.212271 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:00:43.212751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:43.212333 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 21:00:43.585306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:43.585225 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:00:43.585306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:43.585297 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:00:44.162213 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:00:44.162168 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:00:44.212526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:44.212491 5932 patch_prober.go:29] interesting pod/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 21:00:44.212936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:44.212550 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 21:00:45.217634 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:45.217596 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 21:00:45.218895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:45.218872 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 21:00:47.711689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:47.711490 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d"] Nov 03 21:00:47.716603 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:47.716581 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:47.803649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:47.798079 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0776c6-openshift-default-dockercfg-bpxbd\"" Nov 03 21:00:48.310668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.310613 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.310874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.310685 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.310874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.310849 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.311014 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.310891 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.311014 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.310987 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.311498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.311469 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.311793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.311769 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.312006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.311986 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.312178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.312158 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzq77\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.325646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.320413 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d"] Nov 03 21:00:48.414189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.414102 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.416239 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.416209 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.417400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.417377 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.417932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.417914 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.418549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.418530 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.419038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.419019 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.419536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.419517 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.419962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.419947 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.420448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.420429 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzq77\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.420993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.419392 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.427433 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.417864 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.427587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.419901 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.427778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.418943 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.427906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.417175 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.448650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.443367 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.472115 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.472084 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.478831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.478802 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.481178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.481153 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzq77\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77\") pod \"e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:48.700147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:48.700116 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:49.178057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.178017 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d"] Nov 03 21:00:49.182979 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:49.182948 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3071cf01_341c_4548_bdad_5b7b4f0b9d41.slice/crio-dc76f2770f67c8936bf67a24907fd4a3586fd131ba2698d39d26f2778900ee7d WatchSource:0}: Error finding container dc76f2770f67c8936bf67a24907fd4a3586fd131ba2698d39d26f2778900ee7d: Status 404 returned error can't find the container with id dc76f2770f67c8936bf67a24907fd4a3586fd131ba2698d39d26f2778900ee7d Nov 03 21:00:49.185761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.185730 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:49.185843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.185795 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:49.185843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.185822 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:49.358643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.351083 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" event={"ID":"3071cf01-341c-4548-bdad-5b7b4f0b9d41","Type":"ContainerStarted","Data":"f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387"} Nov 03 21:00:49.358643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.351149 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" event={"ID":"3071cf01-341c-4548-bdad-5b7b4f0b9d41","Type":"ContainerStarted","Data":"dc76f2770f67c8936bf67a24907fd4a3586fd131ba2698d39d26f2778900ee7d"} Nov 03 21:00:49.473481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.473380 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podStartSLOduration=2.473366322 podStartE2EDuration="2.473366322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:49.469420321 +0000 UTC m=+1857.534661795" watchObservedRunningTime="2026-11-03 21:00:49.473366322 +0000 UTC m=+1857.538607796" Nov 03 21:00:49.703700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:49.702884 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:50.706393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:50.706363 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:51.384894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:51.384858 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:51.386327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:51.386301 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:00:55.426072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.426038 5932 generic.go:356] "Generic (PLEG): container finished" podID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerID="9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b" exitCode=0 Nov 03 21:00:55.426679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.426112 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" event={"ID":"af9e6dfb-9c04-4894-808f-d8364790e47b","Type":"ContainerDied","Data":"9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b"} Nov 03 21:00:55.426774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.426716 5932 scope.go:122] "RemoveContainer" containerID="9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b" Nov 03 21:00:55.428510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.428483 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:55.428597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.428555 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:55.428597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.428587 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:00:55.827293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:55.827207 5932 scope.go:122] "RemoveContainer" containerID="03ee7826dd4183c4d3ec4d7a034ccd0a01a1ca660d36f6be693aa43d986e9d60" Nov 03 21:00:56.455789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:56.455750 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" event={"ID":"af9e6dfb-9c04-4894-808f-d8364790e47b","Type":"ContainerStarted","Data":"be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65"} Nov 03 21:00:56.468126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:56.468097 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 21:00:56.837704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:56.831989 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 21:00:56.845805 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:56.845752 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 21:00:56.845983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:56.845828 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 21:00:57.849643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:57.847935 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 21:00:57.849643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:57.848007 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 21:00:58.047648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.046185 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh"] Nov 03 21:00:58.081655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.078136 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh"] Nov 03 21:00:58.119921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.108155 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.235296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.204025 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0776c6-dockercfg-zqkjw\"" Nov 03 21:00:58.269646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.258475 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0776c6\"" Nov 03 21:00:58.269646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.269633 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsmcr\" (UniqueName: \"kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr\") pod \"payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.269917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.269777 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.382681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.378686 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsmcr\" (UniqueName: \"kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr\") pod \"payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.394147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.394100 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.396119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.395847 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.477021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.476983 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsmcr\" (UniqueName: \"kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr\") pod \"payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.484550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.484521 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6"] Nov 03 21:00:58.489647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.489610 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6"] Nov 03 21:00:58.489790 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.489710 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.598261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.598226 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.598541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.598510 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmq2k\" (UniqueName: \"kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k\") pod \"payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.645745 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.643906 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:58.700303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.700260 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmq2k\" (UniqueName: \"kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k\") pod \"payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.700955 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.700927 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.703686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.701780 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.762639 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.760306 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmq2k\" (UniqueName: \"kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k\") pod \"payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.807659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.805695 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:58.851826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.851785 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 21:00:58.852285 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:58.851867 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 21:00:59.056049 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:59.056015 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c053419_7452_4329_8ff0_71eff6aa8c02.slice/crio-dc9f7e70f5d7c5656bb1719166f2d257cb8a827c579ccae64767034dcc2facf3 WatchSource:0}: Error finding container dc9f7e70f5d7c5656bb1719166f2d257cb8a827c579ccae64767034dcc2facf3: Status 404 returned error can't find the container with id dc9f7e70f5d7c5656bb1719166f2d257cb8a827c579ccae64767034dcc2facf3 Nov 03 21:00:59.065442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.065371 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh"] Nov 03 21:00:59.253918 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:00:59.253884 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d1bece_24b9_4202_a5c6_285f3e99834a.slice/crio-e409b58a591b260e13da1df9e3c65a20107a8829a6ff9e95978e4bdc3516b93c WatchSource:0}: Error finding container e409b58a591b260e13da1df9e3c65a20107a8829a6ff9e95978e4bdc3516b93c: Status 404 returned error can't find the container with id e409b58a591b260e13da1df9e3c65a20107a8829a6ff9e95978e4bdc3516b93c Nov 03 21:00:59.254668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.254643 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6"] Nov 03 21:00:59.496736 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.496652 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" event={"ID":"2c053419-7452-4329-8ff0-71eff6aa8c02","Type":"ContainerStarted","Data":"00d6605d6fd191b069e82a19e5235e52fd416fec7e4732f66dda50bcd78c6d1d"} Nov 03 21:00:59.496736 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.496708 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" event={"ID":"2c053419-7452-4329-8ff0-71eff6aa8c02","Type":"ContainerStarted","Data":"dc9f7e70f5d7c5656bb1719166f2d257cb8a827c579ccae64767034dcc2facf3"} Nov 03 21:00:59.496736 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.496729 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:00:59.498426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.498401 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" event={"ID":"f2d1bece-24b9-4202-a5c6-285f3e99834a","Type":"ContainerStarted","Data":"63bbc8518e6b12fc792b7bf0669d3ab7fd75ece7ed31adece03753d008b3423c"} Nov 03 21:00:59.498568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.498443 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" event={"ID":"f2d1bece-24b9-4202-a5c6-285f3e99834a","Type":"ContainerStarted","Data":"e409b58a591b260e13da1df9e3c65a20107a8829a6ff9e95978e4bdc3516b93c"} Nov 03 21:00:59.498568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.498493 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:00:59.613065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.613016 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" podStartSLOduration=2.61300182 podStartE2EDuration="2.61300182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:59.6083332 +0000 UTC m=+1867.673574673" watchObservedRunningTime="2026-11-03 21:00:59.61300182 +0000 UTC m=+1867.678243293" Nov 03 21:00:59.780705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.780211 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" podStartSLOduration=2.780172667 podStartE2EDuration="2.780172667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:59.762812664 +0000 UTC m=+1867.828054173" watchObservedRunningTime="2026-11-03 21:00:59.780172667 +0000 UTC m=+1867.845414141" Nov 03 21:00:59.853645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.853170 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 21:00:59.853645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:00:59.853229 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 21:01:00.843280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:00.843242 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 21:01:00.844075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:00.844039 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 21:01:01.845011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:01.844969 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 21:01:01.845534 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:01.845036 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 21:01:02.833270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:02.833238 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 21:01:02.834336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:02.834313 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 21:01:10.512458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:10.512427 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:01:10.512950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:10.512500 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:01:12.489976 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.489940 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c"] Nov 03 21:01:12.495922 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.495901 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.547712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.547679 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d78b90-openshift-default-dockercfg-4x8jk\"" Nov 03 21:01:12.553114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553084 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553239 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553132 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553239 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553175 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgrbw\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553239 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553206 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553372 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553336 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553431 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553375 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553431 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553414 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553561 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553544 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.553705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.553679 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.605393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.605343 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c"] Nov 03 21:01:12.656328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656281 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656444 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656509 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgrbw\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656556 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656596 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656729 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656641 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656729 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656681 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656734 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656781 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.656866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.656856 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.657141 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.657118 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.657226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.657201 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.657310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.657264 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.657900 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.657871 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.660599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.660578 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.660991 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.660971 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.768669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.768559 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgrbw\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.783657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.769433 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token\") pod \"e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:12.817772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:12.813411 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:13.667716 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:13.667678 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c"] Nov 03 21:01:13.670129 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:01:13.670100 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e11e9a_8524_4256_b778_5504f5167180.slice/crio-0e9f7cb54c33ee4c90a9894c383e340046a9e261525b1a8f830e853a553ef517 WatchSource:0}: Error finding container 0e9f7cb54c33ee4c90a9894c383e340046a9e261525b1a8f830e853a553ef517: Status 404 returned error can't find the container with id 0e9f7cb54c33ee4c90a9894c383e340046a9e261525b1a8f830e853a553ef517 Nov 03 21:01:13.673054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:13.673022 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:13.673158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:13.673093 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:13.673158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:13.673123 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:13.754641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:13.753650 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" event={"ID":"c9e11e9a-8524-4256-b778-5504f5167180","Type":"ContainerStarted","Data":"0e9f7cb54c33ee4c90a9894c383e340046a9e261525b1a8f830e853a553ef517"} Nov 03 21:01:14.176428 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:01:14.176395 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:01:14.768686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:14.768555 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" event={"ID":"c9e11e9a-8524-4256-b778-5504f5167180","Type":"ContainerStarted","Data":"0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323"} Nov 03 21:01:14.816788 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:14.816758 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:14.823316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:14.823284 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:14.932128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:14.932062 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podStartSLOduration=2.9320419920000003 podStartE2EDuration="2.932041992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:14.927064178 +0000 UTC m=+1882.992305651" watchObservedRunningTime="2026-11-03 21:01:14.932041992 +0000 UTC m=+1882.997283466" Nov 03 21:01:15.779375 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:15.779339 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:15.780470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:15.780442 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:01:16.835430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:16.835395 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v"] Nov 03 21:01:16.835945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:16.835687 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" containerID="cri-o://8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b" gracePeriod=30 Nov 03 21:01:18.808812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:18.808772 5932 generic.go:356] "Generic (PLEG): container finished" podID="5a834d62-437d-4854-a49d-a23856e72225" containerID="d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d" exitCode=0 Nov 03 21:01:18.809282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:18.808848 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" event={"ID":"5a834d62-437d-4854-a49d-a23856e72225","Type":"ContainerDied","Data":"d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d"} Nov 03 21:01:18.809671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:18.809510 5932 scope.go:122] "RemoveContainer" containerID="d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d" Nov 03 21:01:18.811553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:18.811523 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:18.811666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:18.811593 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:18.811666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:18.811639 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:19.820034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:19.819946 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" event={"ID":"5a834d62-437d-4854-a49d-a23856e72225","Type":"ContainerStarted","Data":"92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9"} Nov 03 21:01:20.753684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:20.753538 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 21:01:20.754901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:20.754872 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 21:01:20.755046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:20.754968 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 21:01:21.652653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.652150 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm"] Nov 03 21:01:21.704707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.704665 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:21.758661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.752994 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r5kc\" (UniqueName: \"kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc\") pod \"payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:21.758661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.753115 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:21.781768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.773994 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 21:01:21.781768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.774053 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 21:01:21.848117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.837981 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d78b90\"" Nov 03 21:01:21.903375 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.851041 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d78b90-dockercfg-v8bmk\"" Nov 03 21:01:21.932650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.904139 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r5kc\" (UniqueName: \"kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc\") pod \"payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:21.956858 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.956812 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:21.969759 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.958367 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:22.060346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.979212 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 21:01:22.060346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.059996 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 21:01:22.060346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.023728 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r5kc\" (UniqueName: \"kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc\") pod \"payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:22.060346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:21.980552 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 21:01:22.105736 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.105670 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm"] Nov 03 21:01:22.114440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.114271 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:22.486526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.486496 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 21:01:22.588770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.588732 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.588770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.588773 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.588989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.588882 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.588989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.588927 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.588989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.588983 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.589183 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589071 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.589183 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589125 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.589306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589237 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.589374 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589323 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c\") pod \"76ffa05d-2883-4928-889d-b17e1350ed4b\" (UID: \"76ffa05d-2883-4928-889d-b17e1350ed4b\") " Nov 03 21:01:22.589374 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589066 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:22.589485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589379 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:22.589485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589165 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:22.589485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589233 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:22.589485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.589461 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:01:22.591436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.591391 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.591565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.591443 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.591565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.591460 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.591565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.591472 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.591565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.591501 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76ffa05d-2883-4928-889d-b17e1350ed4b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.592200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.592171 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:01:22.592330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.592309 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "kube-api-access-fdl6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:22.592500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.592485 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:01:22.592680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.592661 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token" pod "76ffa05d-2883-4928-889d-b17e1350ed4b" (UID: "76ffa05d-2883-4928-889d-b17e1350ed4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:01:22.596238 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.596219 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm"] Nov 03 21:01:22.599523 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:01:22.599499 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48102ead_fefd_47a6_9f72_4ecee9cc7bda.slice/crio-b1593d94a11710e0e7ab098e5f5245b46f07ade17c1a595dd11f2a46cef37a62 WatchSource:0}: Error finding container b1593d94a11710e0e7ab098e5f5245b46f07ade17c1a595dd11f2a46cef37a62: Status 404 returned error can't find the container with id b1593d94a11710e0e7ab098e5f5245b46f07ade17c1a595dd11f2a46cef37a62 Nov 03 21:01:22.693452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.693418 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.693452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.693448 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.693963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.693458 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76ffa05d-2883-4928-889d-b17e1350ed4b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.693963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.693473 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdl6c\" (UniqueName: \"kubernetes.io/projected/76ffa05d-2883-4928-889d-b17e1350ed4b-kube-api-access-fdl6c\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:01:22.761434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.760870 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 21:01:22.761434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:22.760938 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 21:01:23.055706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.055038 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" event={"ID":"48102ead-fefd-47a6-9f72-4ecee9cc7bda","Type":"ContainerStarted","Data":"e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437"} Nov 03 21:01:23.055706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.055107 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" event={"ID":"48102ead-fefd-47a6-9f72-4ecee9cc7bda","Type":"ContainerStarted","Data":"b1593d94a11710e0e7ab098e5f5245b46f07ade17c1a595dd11f2a46cef37a62"} Nov 03 21:01:23.055706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.055167 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:23.060676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.057665 5932 generic.go:356] "Generic (PLEG): container finished" podID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerID="8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b" exitCode=0 Nov 03 21:01:23.060676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.057762 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" Nov 03 21:01:23.060676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.057988 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" event={"ID":"76ffa05d-2883-4928-889d-b17e1350ed4b","Type":"ContainerDied","Data":"8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b"} Nov 03 21:01:23.060676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.058026 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v" event={"ID":"76ffa05d-2883-4928-889d-b17e1350ed4b","Type":"ContainerDied","Data":"36cf48c351cd3060b6a25228511c9acc150920c2ef11f6a036badf0a2456cb93"} Nov 03 21:01:23.060676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.058054 5932 scope.go:122] "RemoveContainer" containerID="8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b" Nov 03 21:01:23.071848 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.071773 5932 scope.go:122] "RemoveContainer" containerID="8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b" Nov 03 21:01:23.072137 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:01:23.072116 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b\": container with ID starting with 8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b not found: ID does not exist" containerID="8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b" Nov 03 21:01:23.072189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.072147 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b"} err="failed to get container status \"8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b\": rpc error: code = NotFound desc = could not find container \"8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b\": container with ID starting with 8904c50d06d4c20f9e236d958e7e945746858747d237ea097e84a499a28b250b not found: ID does not exist" Nov 03 21:01:23.357711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.349655 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" podStartSLOduration=2.349613418 podStartE2EDuration="2.349613418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:23.180959986 +0000 UTC m=+1891.246201472" watchObservedRunningTime="2026-11-03 21:01:23.349613418 +0000 UTC m=+1891.414854892" Nov 03 21:01:23.366700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.359787 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v"] Nov 03 21:01:23.408648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.388523 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-zcm9v"] Nov 03 21:01:23.754640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.754018 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 21:01:23.754640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:23.754086 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 21:01:24.083908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:24.083704 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" path="/var/lib/kubelet/pods/76ffa05d-2883-4928-889d-b17e1350ed4b/volumes" Nov 03 21:01:24.764676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:24.758379 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 21:01:24.764676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:24.758457 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 21:01:25.753945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:25.753911 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 21:01:25.754138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:25.753966 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 21:01:26.759696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:26.759659 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 21:01:26.761107 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:26.761077 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 21:01:31.921416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:31.921377 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2"] Nov 03 21:01:31.922553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:31.922527 5932 cpu_manager.go:414] "containerMap: removing container" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" Nov 03 21:01:31.922726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:31.922554 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" Nov 03 21:01:31.922823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:31.922807 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ffa05d-2883-4928-889d-b17e1350ed4b" containerName="istio-proxy" Nov 03 21:01:31.928493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:31.928470 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:31.981414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:31.981379 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2"] Nov 03 21:01:32.009087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.009049 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpt4j\" (UniqueName: \"kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.009323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.009123 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.009323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.009257 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.009469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.009430 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.064912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.064878 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-af2e8a-serving-cert\"" Nov 03 21:01:32.110709 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.110664 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.110889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.110757 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.110889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.110863 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.111025 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:01:32.111005 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-serving-cert: secret "maas-api-e2e-api-a-af2e8a-serving-cert" not found Nov 03 21:01:32.111099 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.111009 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt4j\" (UniqueName: \"kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.111162 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:01:32.111088 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls podName:8020df72-b51d-4acb-9816-46ee6d01ce6c nodeName:}" failed. No retries permitted until 2026-11-03 21:01:32.611070759 +0000 UTC m=+1900.676312210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls") pod "maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" (UID: "8020df72-b51d-4acb-9816-46ee6d01ce6c") : secret "maas-api-e2e-api-a-af2e8a-serving-cert" not found Nov 03 21:01:32.111460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.111439 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.113296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.113272 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.193774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.183038 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt4j\" (UniqueName: \"kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.627487 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.627455 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.630196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.630167 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\") pod \"maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:32.844147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:32.844072 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:33.556853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.556738 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98"] Nov 03 21:01:33.629278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.629179 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98"] Nov 03 21:01:33.629470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.629294 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.634802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.634753 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfsk\" (UniqueName: \"kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk\") pod \"payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.634990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.634850 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.678870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.678840 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-8448c9\"" Nov 03 21:01:33.679082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.678908 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-8448c9-dockercfg-c9njg\"" Nov 03 21:01:33.735855 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.735825 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfsk\" (UniqueName: \"kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk\") pod \"payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.736030 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.735935 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.737072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.736867 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.794836 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.794803 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfsk\" (UniqueName: \"kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk\") pod \"payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:33.906171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.906134 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2"] Nov 03 21:01:33.910857 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:01:33.910774 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8020df72_b51d_4acb_9816_46ee6d01ce6c.slice/crio-8c550024e5ee3a55c71ea061b73bf22d82a8f2c9329df995376a354998c0ce07 WatchSource:0}: Error finding container 8c550024e5ee3a55c71ea061b73bf22d82a8f2c9329df995376a354998c0ce07: Status 404 returned error can't find the container with id 8c550024e5ee3a55c71ea061b73bf22d82a8f2c9329df995376a354998c0ce07 Nov 03 21:01:33.953733 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:33.953696 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:34.085318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:34.085282 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:01:34.304898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:34.304809 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" event={"ID":"8020df72-b51d-4acb-9816-46ee6d01ce6c","Type":"ContainerStarted","Data":"8c550024e5ee3a55c71ea061b73bf22d82a8f2c9329df995376a354998c0ce07"} Nov 03 21:01:34.410663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:34.410591 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98"] Nov 03 21:01:34.412424 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:01:34.412373 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152dd512_4484_489e_b01b_53ca4d512db6.slice/crio-d7a6144bd5764b6b96db7baa37d873565f46a39f0db83d3ec6b5c62dd240e5e2 WatchSource:0}: Error finding container d7a6144bd5764b6b96db7baa37d873565f46a39f0db83d3ec6b5c62dd240e5e2: Status 404 returned error can't find the container with id d7a6144bd5764b6b96db7baa37d873565f46a39f0db83d3ec6b5c62dd240e5e2 Nov 03 21:01:35.359931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.359424 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" event={"ID":"152dd512-4484-489e-b01b-53ca4d512db6","Type":"ContainerStarted","Data":"e0f3df577769177c362d4880d8da4cd5bf23a42aa127e8c37948eb078f2f9d97"} Nov 03 21:01:35.359931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.359493 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:35.359931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.359515 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" event={"ID":"152dd512-4484-489e-b01b-53ca4d512db6","Type":"ContainerStarted","Data":"d7a6144bd5764b6b96db7baa37d873565f46a39f0db83d3ec6b5c62dd240e5e2"} Nov 03 21:01:35.362215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.362186 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" event={"ID":"8020df72-b51d-4acb-9816-46ee6d01ce6c","Type":"ContainerStarted","Data":"389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01"} Nov 03 21:01:35.362344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.362273 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:35.487387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.487335 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" podStartSLOduration=2.487317696 podStartE2EDuration="2.487317696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:35.480052092 +0000 UTC m=+1903.545293566" watchObservedRunningTime="2026-11-03 21:01:35.487317696 +0000 UTC m=+1903.552559168" Nov 03 21:01:35.602378 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:35.602308 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" podStartSLOduration=3.720317082 podStartE2EDuration="4.602237739s" totalImagesPullingTime="881.920657ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:01:35.592298024 +0000 UTC m=+1903.657539490" watchObservedRunningTime="2026-11-03 21:01:35.602237739 +0000 UTC m=+1903.667479213" Nov 03 21:01:41.377701 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:41.377669 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:01:44.153006 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:01:44.152968 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:01:46.374754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:46.374722 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:01:55.785795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.785756 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:55.786246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.785816 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:55.786776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.786736 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:01:55.786910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.786787 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:01:55.932647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.932597 5932 scope.go:122] "RemoveContainer" containerID="32c2b2d696168bcf020c5794ba0aec9eab9e9643b329780fb6144b98fae06247" Nov 03 21:01:55.948294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.948233 5932 scope.go:122] "RemoveContainer" containerID="66649da2eff3f903a465fd436067e61e8066c9f1ad71eb040ed6f4111a496b29" Nov 03 21:01:55.962938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.962906 5932 scope.go:122] "RemoveContainer" containerID="26969f472eac18a47c8dedf9171d5ab0350ca883c27c42a872ee74122e5dc36b" Nov 03 21:01:55.980970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:55.980943 5932 scope.go:122] "RemoveContainer" containerID="4c8e69f91117c583a1a3929f93934b3ca99877c8969c7a8441f60d6da90143e9" Nov 03 21:01:58.713128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.713092 5932 generic.go:356] "Generic (PLEG): container finished" podID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerID="6792df5727ef0b3ef240dfcc45b089ce09a4b2e18edd1c066a212bef477ae7c9" exitCode=0 Nov 03 21:01:58.713707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.713167 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" event={"ID":"a847dd75-4df4-45dd-8511-7bde98fd7c92","Type":"ContainerDied","Data":"6792df5727ef0b3ef240dfcc45b089ce09a4b2e18edd1c066a212bef477ae7c9"} Nov 03 21:01:58.713798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.713778 5932 scope.go:122] "RemoveContainer" containerID="6792df5727ef0b3ef240dfcc45b089ce09a4b2e18edd1c066a212bef477ae7c9" Nov 03 21:01:58.715356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.715334 5932 generic.go:356] "Generic (PLEG): container finished" podID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerID="0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47" exitCode=0 Nov 03 21:01:58.715482 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.715382 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" event={"ID":"80d57f71-663f-4a44-a0c7-59db19c7ad1e","Type":"ContainerDied","Data":"0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47"} Nov 03 21:01:58.715860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.715834 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:58.715954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.715873 5932 scope.go:122] "RemoveContainer" containerID="0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47" Nov 03 21:01:58.715954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.715901 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:58.715954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.715941 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:58.717574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.717544 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:58.717691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.717610 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:58.717691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:58.717663 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:01:59.775647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:59.774502 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" event={"ID":"a847dd75-4df4-45dd-8511-7bde98fd7c92","Type":"ContainerStarted","Data":"b9247ca5c52722e2541626e366ad1620bb45b0e6780144831bf134eae7dcb03e"} Nov 03 21:01:59.783791 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:59.783703 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" event={"ID":"80d57f71-663f-4a44-a0c7-59db19c7ad1e","Type":"ContainerStarted","Data":"3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8"} Nov 03 21:01:59.892477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:59.892430 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:01:59.892827 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:59.892805 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 21:01:59.892886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:01:59.892855 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 21:02:00.346395 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:00.345875 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:02:00.346395 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:00.346031 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 21:02:00.346395 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:00.346079 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 21:02:00.892793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:00.892755 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 21:02:00.893194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:00.892821 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 21:02:01.362705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:01.360098 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 21:02:01.362705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:01.360162 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 21:02:01.892426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:01.892389 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 21:02:01.892668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:01.892453 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 21:02:02.348703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:02.347765 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 21:02:02.348703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:02.347838 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 21:02:02.892789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:02.892751 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 21:02:02.892994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:02.892813 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 21:02:03.360707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:03.360194 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 21:02:03.360707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:03.360255 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 21:02:03.892334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:03.892295 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 21:02:03.892608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:03.892358 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 21:02:04.363703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.360809 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 21:02:04.363703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.360874 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 21:02:04.887194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.887159 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r"] Nov 03 21:02:04.887421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.887370 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" containerID="cri-o://fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042" gracePeriod=30 Nov 03 21:02:04.892570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.892546 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body= Nov 03 21:02:04.892698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.892590 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" Nov 03 21:02:04.974281 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.973743 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j"] Nov 03 21:02:04.974281 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.974016 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" containerID="cri-o://f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673" gracePeriod=30 Nov 03 21:02:04.979128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.979104 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6"] Nov 03 21:02:04.979493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.979473 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" containerID="cri-o://9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06" gracePeriod=30 Nov 03 21:02:04.983477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.983451 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7"] Nov 03 21:02:04.983675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:04.983654 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" containerID="cri-o://9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1" gracePeriod=30 Nov 03 21:02:05.350894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:05.350795 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body= Nov 03 21:02:05.350894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:05.350872 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" Nov 03 21:02:05.903104 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:05.901886 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:02:05.903665 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:05.903245 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:02:05.904323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:05.904294 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:02:05.984644 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:05.982097 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9"] Nov 03 21:02:06.011650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.011394 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.078658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.070354 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-878695\"" Nov 03 21:02:06.078658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.070484 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-878695-dockercfg-kd592\"" Nov 03 21:02:06.227696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.146312 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.227696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.146386 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z\") pod \"payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.249458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.249410 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z\") pod \"payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.250737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.250713 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.251347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.251320 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9"] Nov 03 21:02:06.251535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.251516 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.308690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.308659 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z\") pod \"payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.350689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.350658 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:02:06.350970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.350952 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:02:06.351748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.351727 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:02:06.548795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.548710 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:06.940911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:06.940875 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9"] Nov 03 21:02:06.945101 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:02:06.945075 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0985b16e_aac5_4ceb_80a3_ea2a54800271.slice/crio-90b3b7cd2264c71227279b811652c4f9871b177c6645ef9fbc50fcbd8aacd027 WatchSource:0}: Error finding container 90b3b7cd2264c71227279b811652c4f9871b177c6645ef9fbc50fcbd8aacd027: Status 404 returned error can't find the container with id 90b3b7cd2264c71227279b811652c4f9871b177c6645ef9fbc50fcbd8aacd027 Nov 03 21:02:07.921111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:07.921072 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" event={"ID":"0985b16e-aac5-4ceb-80a3-ea2a54800271","Type":"ContainerStarted","Data":"71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef"} Nov 03 21:02:07.921302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:07.921205 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:07.921302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:07.921247 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" event={"ID":"0985b16e-aac5-4ceb-80a3-ea2a54800271","Type":"ContainerStarted","Data":"90b3b7cd2264c71227279b811652c4f9871b177c6645ef9fbc50fcbd8aacd027"} Nov 03 21:02:08.034442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:08.034384 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" podStartSLOduration=3.034370473 podStartE2EDuration="3.034370473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:08.029417136 +0000 UTC m=+1936.094658606" watchObservedRunningTime="2026-11-03 21:02:08.034370473 +0000 UTC m=+1936.099611939" Nov 03 21:02:09.986016 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:09.985972 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbd4e68_62c9_4fde_b3f3_ca809cb6f13c.slice/crio-conmon-fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbd4e68_62c9_4fde_b3f3_ca809cb6f13c.slice/crio-fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:02:10.197904 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.197496 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 21:02:10.291230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.291209 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.348885 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349028 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349076 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349126 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349316 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349357 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349391 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.349532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.349468 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.350460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.350278 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy\") pod \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\" (UID: \"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86\") " Nov 03 21:02:10.353610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.352535 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.353610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.353258 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:10.353610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.353385 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.353610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.353577 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.354146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.354126 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.356657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.356602 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:10.357722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.357698 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.363909 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.363889 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.366551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.366527 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp" pod "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" (UID: "0e448d6c-a692-4edf-9ac5-43ad4bb8dc86"). InnerVolumeSpecName "kube-api-access-tz5wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.413842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.413809 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 21:02:10.427556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.427479 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 21:02:10.452508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.452475 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.452706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.452594 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.452799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.452722 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.452799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.452771 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.452929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.452808 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.453005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.452987 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.453082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453056 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.453148 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453091 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.453207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453147 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.453207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453179 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.453315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453233 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.453315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453262 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.453438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453342 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.453438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453406 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.453541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453455 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket\") pod \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\" (UID: \"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c\") " Nov 03 21:02:10.453541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453455 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:10.453931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453906 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.454247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.454223 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.454337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.454262 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.454448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.454413 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.454517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.454489 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.454661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.453931 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:10.454815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.454797 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.454923 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.454907 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz5wp\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-kube-api-access-tz5wp\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455018 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455127 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455114 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455218 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455343 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455450 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455578 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455727 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455838 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455826 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.455937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455925 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.456021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.455352 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.456324 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.456294 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.460658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.460596 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:10.460658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.460651 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "kube-api-access-9k8qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.460826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.460703 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "kube-api-access-tph6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.460826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.460609 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:10.460826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.460783 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.462415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.462394 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token" pod "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" (UID: "abbd4e68-62c9-4fde-b3f3-ca809cb6f13c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.557437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557399 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.557674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557468 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.557674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557494 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.557674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557533 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.557674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557643 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557684 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557742 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557776 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557817 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557856 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557859 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557874 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557926 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket\") pod \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\" (UID: \"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557926 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557957 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token\") pod \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\" (UID: \"81f419fb-6c7a-4922-9f0c-ceb825172fd2\") " Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.557952 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.558159 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.558165 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.558774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.558360 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559027 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559054 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559070 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559086 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559102 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559116 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559130 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559144 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559158 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tph6p\" (UniqueName: \"kubernetes.io/projected/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-kube-api-access-tph6p\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559173 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559188 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559205 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559218 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559232 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559246 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559259 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559274 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k8qj\" (UniqueName: \"kubernetes.io/projected/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c-kube-api-access-9k8qj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559287 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559300 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.559540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.559314 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.560563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.560532 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:10.560652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.560563 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.560726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.560656 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy" pod "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" (UID: "e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:10.560726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.560672 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "kube-api-access-vgnqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.560820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.560805 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token" pod "81f419fb-6c7a-4922-9f0c-ceb825172fd2" (UID: "81f419fb-6c7a-4922-9f0c-ceb825172fd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:10.661133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.661095 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgnqg\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-kube-api-access-vgnqg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.661133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.661127 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.661133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.661138 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.661460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.661150 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.661460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.661165 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f419fb-6c7a-4922-9f0c-ceb825172fd2-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:10.950756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.950721 5932 generic.go:356] "Generic (PLEG): container finished" podID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerID="fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042" exitCode=0 Nov 03 21:02:10.950919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.950789 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" Nov 03 21:02:10.950919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.950796 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" event={"ID":"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c","Type":"ContainerDied","Data":"fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042"} Nov 03 21:02:10.950919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.950873 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r" event={"ID":"abbd4e68-62c9-4fde-b3f3-ca809cb6f13c","Type":"ContainerDied","Data":"a9a9780ffadfc88e3b5a2f1148b246e59c12802e54557fee876771352f248ec6"} Nov 03 21:02:10.950919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.950896 5932 scope.go:122] "RemoveContainer" containerID="fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042" Nov 03 21:02:10.952907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.952881 5932 generic.go:356] "Generic (PLEG): container finished" podID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerID="9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1" exitCode=0 Nov 03 21:02:10.953048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.952942 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" Nov 03 21:02:10.953048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.952960 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" event={"ID":"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017","Type":"ContainerDied","Data":"9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1"} Nov 03 21:02:10.953048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.953007 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7" event={"ID":"e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017","Type":"ContainerDied","Data":"ddad01fcc6322ba3bfd855abc1d795e1c1a7a10290d0c95125934a09c94b38d0"} Nov 03 21:02:10.954975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.954950 5932 generic.go:356] "Generic (PLEG): container finished" podID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerID="f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673" exitCode=0 Nov 03 21:02:10.955108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.955005 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" Nov 03 21:02:10.955108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.955032 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" event={"ID":"81f419fb-6c7a-4922-9f0c-ceb825172fd2","Type":"ContainerDied","Data":"f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673"} Nov 03 21:02:10.955108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.955096 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j" event={"ID":"81f419fb-6c7a-4922-9f0c-ceb825172fd2","Type":"ContainerDied","Data":"5f00d365a569e75573e2226f5b05c64cd81ee95c36078a50d5f28a7767c571fd"} Nov 03 21:02:10.956984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.956877 5932 generic.go:356] "Generic (PLEG): container finished" podID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerID="9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06" exitCode=0 Nov 03 21:02:10.956984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.956916 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" event={"ID":"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86","Type":"ContainerDied","Data":"9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06"} Nov 03 21:02:10.956984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.956954 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" event={"ID":"0e448d6c-a692-4edf-9ac5-43ad4bb8dc86","Type":"ContainerDied","Data":"1a364074fafbc69031e31ce0a27681b43b78fbfd3b198d83929d3aa385a20bea"} Nov 03 21:02:10.956984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.956974 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6" Nov 03 21:02:10.971436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.971418 5932 scope.go:122] "RemoveContainer" containerID="fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042" Nov 03 21:02:10.971787 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:10.971767 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042\": container with ID starting with fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042 not found: ID does not exist" containerID="fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042" Nov 03 21:02:10.971863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.971793 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042"} err="failed to get container status \"fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042\": rpc error: code = NotFound desc = could not find container \"fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042\": container with ID starting with fa3fe6061c9ab00ca278153016e07e632ee746103c16d66019eab788765ce042 not found: ID does not exist" Nov 03 21:02:10.971863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.971810 5932 scope.go:122] "RemoveContainer" containerID="9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1" Nov 03 21:02:10.982357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.982338 5932 scope.go:122] "RemoveContainer" containerID="8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89" Nov 03 21:02:10.992167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.992144 5932 scope.go:122] "RemoveContainer" containerID="9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1" Nov 03 21:02:10.992462 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:10.992442 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1\": container with ID starting with 9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1 not found: ID does not exist" containerID="9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1" Nov 03 21:02:10.992508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.992469 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1"} err="failed to get container status \"9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1\": rpc error: code = NotFound desc = could not find container \"9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1\": container with ID starting with 9c7fd6bd139c9e3522e3171521fc92c662aaa1983f3ec9358c82cbddb6c9bbe1 not found: ID does not exist" Nov 03 21:02:10.992508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.992486 5932 scope.go:122] "RemoveContainer" containerID="8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89" Nov 03 21:02:10.992750 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:10.992718 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89\": container with ID starting with 8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89 not found: ID does not exist" containerID="8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89" Nov 03 21:02:10.992801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.992757 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89"} err="failed to get container status \"8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89\": rpc error: code = NotFound desc = could not find container \"8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89\": container with ID starting with 8172a3568466999a226a3ce50ec4bf3d12aab77ce68461feff1623e6942adb89 not found: ID does not exist" Nov 03 21:02:10.992801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:10.992774 5932 scope.go:122] "RemoveContainer" containerID="f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673" Nov 03 21:02:11.003062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.002933 5932 scope.go:122] "RemoveContainer" containerID="f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673" Nov 03 21:02:11.003223 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:11.003201 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673\": container with ID starting with f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673 not found: ID does not exist" containerID="f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673" Nov 03 21:02:11.003265 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.003229 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673"} err="failed to get container status \"f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673\": rpc error: code = NotFound desc = could not find container \"f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673\": container with ID starting with f4c53bb785100c19fdc7fb3fcb5cda2d3bb616fef9191c2751ff5d1e303a1673 not found: ID does not exist" Nov 03 21:02:11.003265 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.003252 5932 scope.go:122] "RemoveContainer" containerID="9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06" Nov 03 21:02:11.014350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.014332 5932 scope.go:122] "RemoveContainer" containerID="9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06" Nov 03 21:02:11.014680 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:11.014660 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06\": container with ID starting with 9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06 not found: ID does not exist" containerID="9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06" Nov 03 21:02:11.014751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.014688 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06"} err="failed to get container status \"9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06\": rpc error: code = NotFound desc = could not find container \"9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06\": container with ID starting with 9454ca4bc283009e68b55dfd7258784356dfcd9e2a71bbaf4c5de92584f5ae06 not found: ID does not exist" Nov 03 21:02:11.107603 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.107569 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7"] Nov 03 21:02:11.153416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.153382 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lcsc7"] Nov 03 21:02:11.213571 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.213487 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j"] Nov 03 21:02:11.218596 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.218559 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-66t8j"] Nov 03 21:02:11.320481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.320449 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r"] Nov 03 21:02:11.329436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.329399 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-vml9r"] Nov 03 21:02:11.430808 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.430770 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6"] Nov 03 21:02:11.436633 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:11.436594 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-njdm6"] Nov 03 21:02:12.054760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:12.054729 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" path="/var/lib/kubelet/pods/0e448d6c-a692-4edf-9ac5-43ad4bb8dc86/volumes" Nov 03 21:02:12.055222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:12.055207 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" path="/var/lib/kubelet/pods/81f419fb-6c7a-4922-9f0c-ceb825172fd2/volumes" Nov 03 21:02:12.055734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:12.055719 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" path="/var/lib/kubelet/pods/abbd4e68-62c9-4fde-b3f3-ca809cb6f13c/volumes" Nov 03 21:02:12.056537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:12.056521 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" path="/var/lib/kubelet/pods/e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017/volumes" Nov 03 21:02:13.527542 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:13.527496 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4"] Nov 03 21:02:13.528456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:13.528390 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" containerID="cri-o://42564e5a06712a7e210bbacc7c689318084dbab6474c7ea892f821b9a9bf66fc" gracePeriod=30 Nov 03 21:02:14.150754 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:14.150720 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:02:18.248530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.248498 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g"] Nov 03 21:02:18.249331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249316 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" Nov 03 21:02:18.249331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249332 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249351 5932 cpu_manager.go:414] "containerMap: removing container" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249356 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249370 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249376 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249392 5932 cpu_manager.go:414] "containerMap: removing container" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249397 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249422 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" Nov 03 21:02:18.249432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.249427 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" Nov 03 21:02:18.250327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.250304 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e448d6c-a692-4edf-9ac5-43ad4bb8dc86" containerName="istio-proxy" Nov 03 21:02:18.250412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.250356 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" Nov 03 21:02:18.250412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.250387 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f419fb-6c7a-4922-9f0c-ceb825172fd2" containerName="istio-proxy" Nov 03 21:02:18.250412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.250406 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74ac3d3-f1ab-4e5d-9074-b1ad16ef8017" containerName="istio-proxy" Nov 03 21:02:18.250559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.250428 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbd4e68-62c9-4fde-b3f3-ca809cb6f13c" containerName="istio-proxy" Nov 03 21:02:18.255253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.255229 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.308335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.308308 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g"] Nov 03 21:02:18.349894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.349863 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0776c6-serving-cert\"" Nov 03 21:02:18.385053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.385014 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.385216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.385194 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2vc\" (UniqueName: \"kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.385362 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.385339 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.385422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.385373 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.515509 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.489462 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.516496 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.516456 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.516954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.516933 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.517272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.517255 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2vc\" (UniqueName: \"kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.523232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.507933 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.523232 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:18.522772 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-serving-cert: secret "maas-api-e2e-api-b-0776c6-serving-cert" not found Nov 03 21:02:18.523232 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:18.522860 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls podName:49438490-df7e-4267-90af-57c0b9394625 nodeName:}" failed. No retries permitted until 2026-11-03 21:02:19.022837384 +0000 UTC m=+1947.088078850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls") pod "maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" (UID: "49438490-df7e-4267-90af-57c0b9394625") : secret "maas-api-e2e-api-b-0776c6-serving-cert" not found Nov 03 21:02:18.523232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.523178 5932 patch_prober.go:29] interesting pod/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:02:18.523232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.523217 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:02:18.559846 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.555334 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.575717 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.575688 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2vc\" (UniqueName: \"kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:18.931929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:18.931897 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:02:19.029630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.029599 5932 generic.go:356] "Generic (PLEG): container finished" podID="d768da6e-fcd6-4389-babb-f662d3763675" containerID="42564e5a06712a7e210bbacc7c689318084dbab6474c7ea892f821b9a9bf66fc" exitCode=0 Nov 03 21:02:19.029792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.029672 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" event={"ID":"d768da6e-fcd6-4389-babb-f662d3763675","Type":"ContainerDied","Data":"42564e5a06712a7e210bbacc7c689318084dbab6474c7ea892f821b9a9bf66fc"} Nov 03 21:02:19.034516 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.034489 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:19.037037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.037013 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\") pod \"maas-api-e2e-api-b-0776c6-67c664544c-bgz4g\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:19.176711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.175608 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:19.237640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.237603 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 21:02:19.338456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338420 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338475 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338543 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338567 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338690 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338735 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338765 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338845 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338902 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338935 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:19.339008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.338968 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert\") pod \"d768da6e-fcd6-4389-babb-f662d3763675\" (UID: \"d768da6e-fcd6-4389-babb-f662d3763675\") " Nov 03 21:02:19.339527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.339027 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:19.339527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.339044 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:19.339527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.339306 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:19.341100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341068 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d768da6e-fcd6-4389-babb-f662d3763675-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.341100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341097 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.341268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341113 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.341268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341126 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.341268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341140 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.341756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341736 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:19.341842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341772 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:02:19.341842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341792 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "kube-api-access-2f7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:19.341842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.341784 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy" pod "d768da6e-fcd6-4389-babb-f662d3763675" (UID: "d768da6e-fcd6-4389-babb-f662d3763675"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:02:19.442772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.442695 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d768da6e-fcd6-4389-babb-f662d3763675-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.442772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.442724 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.442772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.442735 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d768da6e-fcd6-4389-babb-f662d3763675-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.442772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.442745 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f7jk\" (UniqueName: \"kubernetes.io/projected/d768da6e-fcd6-4389-babb-f662d3763675-kube-api-access-2f7jk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:19.582304 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:19.582271 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g"] Nov 03 21:02:19.586870 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:02:19.586836 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49438490_df7e_4267_90af_57c0b9394625.slice/crio-d1813e66769d64af669f03bb0d8cabf88243c8f8cb2f09535683b9fc6b8413a3 WatchSource:0}: Error finding container d1813e66769d64af669f03bb0d8cabf88243c8f8cb2f09535683b9fc6b8413a3: Status 404 returned error can't find the container with id d1813e66769d64af669f03bb0d8cabf88243c8f8cb2f09535683b9fc6b8413a3 Nov 03 21:02:20.039916 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:20.039881 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" event={"ID":"49438490-df7e-4267-90af-57c0b9394625","Type":"ContainerStarted","Data":"d1813e66769d64af669f03bb0d8cabf88243c8f8cb2f09535683b9fc6b8413a3"} Nov 03 21:02:20.041400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:20.041369 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" Nov 03 21:02:20.041523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:20.041397 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4" event={"ID":"d768da6e-fcd6-4389-babb-f662d3763675","Type":"ContainerDied","Data":"0fad13ebcca4ceaf598de82b253635b466db7797a84fe53701eb2787b2311c9a"} Nov 03 21:02:20.041523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:20.041464 5932 scope.go:122] "RemoveContainer" containerID="42564e5a06712a7e210bbacc7c689318084dbab6474c7ea892f821b9a9bf66fc" Nov 03 21:02:20.197618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:20.197585 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4"] Nov 03 21:02:20.244549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:20.244516 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-6v2n4"] Nov 03 21:02:21.061443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:21.061403 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" event={"ID":"49438490-df7e-4267-90af-57c0b9394625","Type":"ContainerStarted","Data":"f82979e9e3b13cdb3f2ac9e69a99d9edaf26332f3ea82ec53eef1ca8a1a46478"} Nov 03 21:02:21.061983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:21.061632 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:21.179653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:21.179555 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" podStartSLOduration=2.270466849 podStartE2EDuration="3.179536474s" totalImagesPullingTime="909.069625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:21.17896075 +0000 UTC m=+1949.244202222" watchObservedRunningTime="2026-11-03 21:02:21.179536474 +0000 UTC m=+1949.244777951" Nov 03 21:02:22.054005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:22.053971 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d768da6e-fcd6-4389-babb-f662d3763675" path="/var/lib/kubelet/pods/d768da6e-fcd6-4389-babb-f662d3763675/volumes" Nov 03 21:02:27.076831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:27.076795 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:02:41.191000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.190967 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6927-ff5793"] Nov 03 21:02:41.191828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.191811 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" Nov 03 21:02:41.191828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.191828 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" Nov 03 21:02:41.191973 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.191964 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d768da6e-fcd6-4389-babb-f662d3763675" containerName="istio-proxy" Nov 03 21:02:41.197296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.197273 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:41.246849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.246818 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6927-ff5793"] Nov 03 21:02:41.287723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.287696 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-h2jkh\"" Nov 03 21:02:41.308781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.308751 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt92j\" (UniqueName: \"kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j\") pod \"test-curl-6927-ff5793\" (UID: \"320fcf13-9bc0-412c-acf9-52b8ce758f10\") " pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:41.410785 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.410749 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt92j\" (UniqueName: \"kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j\") pod \"test-curl-6927-ff5793\" (UID: \"320fcf13-9bc0-412c-acf9-52b8ce758f10\") " pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:41.468236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.468159 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt92j\" (UniqueName: \"kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j\") pod \"test-curl-6927-ff5793\" (UID: \"320fcf13-9bc0-412c-acf9-52b8ce758f10\") " pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:41.510823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.510795 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:41.886136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:41.886103 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6927-ff5793"] Nov 03 21:02:41.891206 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:02:41.891171 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod320fcf13_9bc0_412c_acf9_52b8ce758f10.slice/crio-79bdeceab52ebd5c64e44909abcc60e867e4e3c6fcb75a8257a127c55882b8c7 WatchSource:0}: Error finding container 79bdeceab52ebd5c64e44909abcc60e867e4e3c6fcb75a8257a127c55882b8c7: Status 404 returned error can't find the container with id 79bdeceab52ebd5c64e44909abcc60e867e4e3c6fcb75a8257a127c55882b8c7 Nov 03 21:02:42.242933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:42.242853 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6927-ff5793" event={"ID":"320fcf13-9bc0-412c-acf9-52b8ce758f10","Type":"ContainerStarted","Data":"79bdeceab52ebd5c64e44909abcc60e867e4e3c6fcb75a8257a127c55882b8c7"} Nov 03 21:02:44.158836 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:44.158800 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:02:45.274249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:45.274214 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6927-ff5793" event={"ID":"320fcf13-9bc0-412c-acf9-52b8ce758f10","Type":"ContainerStarted","Data":"0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0"} Nov 03 21:02:45.384693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:45.384634 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6927-ff5793" podStartSLOduration=1.6267567889999999 podStartE2EDuration="4.384609158s" totalImagesPullingTime="2.757852369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:45.380925706 +0000 UTC m=+1973.446167161" watchObservedRunningTime="2026-11-03 21:02:45.384609158 +0000 UTC m=+1973.449850631" Nov 03 21:02:46.873284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:46.873246 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6927-ff5793"] Nov 03 21:02:46.877917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:46.877888 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6927-ff5793"] Nov 03 21:02:46.997897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:46.997855 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg"] Nov 03 21:02:46.998783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:46.998758 5932 cpu_manager.go:414] "containerMap: removing container" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" containerName="test-curl-6927-ff5793" Nov 03 21:02:46.998783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:46.998782 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" containerName="test-curl-6927-ff5793" Nov 03 21:02:46.999051 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:46.999025 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" containerName="test-curl-6927-ff5793" Nov 03 21:02:47.002954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.002934 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.058025 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.057995 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg"] Nov 03 21:02:47.098443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.098411 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d78b90-serving-cert\"" Nov 03 21:02:47.099257 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.099225 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.099389 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.099363 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.099557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.099537 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl629\" (UniqueName: \"kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.099650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.099613 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.201451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.201409 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl629\" (UniqueName: \"kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.201681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.201480 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.201766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.201737 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.201872 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.201853 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.202150 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:47.202125 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-serving-cert: secret "maas-api-e2e-shared-b-w2-d78b90-serving-cert" not found Nov 03 21:02:47.202267 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:47.202202 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls podName:6a22bcae-cd8d-4522-bf24-0392d534bfc9 nodeName:}" failed. No retries permitted until 2026-11-03 21:02:47.702180118 +0000 UTC m=+1975.767421593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" (UID: "6a22bcae-cd8d-4522-bf24-0392d534bfc9") : secret "maas-api-e2e-shared-b-w2-d78b90-serving-cert" not found Nov 03 21:02:47.202922 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.202897 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.204909 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.204887 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.283983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.283947 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl629\" (UniqueName: \"kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.312643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.310455 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6927-ff5793" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" containerName="test-curl-6927-ff5793" containerID="cri-o://0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0" gracePeriod=2 Nov 03 21:02:47.372650 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:47.360939 5932 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6927-ff5793\" is forbidden: User \"system:node:ip-10-0-138-52.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-52.us-west-2.compute.internal' and this object" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:47.724672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.724618 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.727362 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.727333 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.884731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.884689 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46"] Nov 03 21:02:47.889616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.889595 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:47.917536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.917501 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:47.938615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.938577 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:02:47.938794 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.938766 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:02:47.963041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:47.943257 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46"] Nov 03 21:02:48.030301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.029871 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.030301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.029910 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7dvm\" (UniqueName: \"kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.030301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.030089 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.043575 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:48.043532 5932 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6927-ff5793\" is forbidden: User \"system:node:ip-10-0-138-52.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-52.us-west-2.compute.internal' and this object" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:48.131697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.131617 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.131875 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.131713 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7dvm\" (UniqueName: \"kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.131875 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.131836 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.132225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.132205 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.134241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.134218 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.189160 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.189130 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7dvm\" (UniqueName: \"kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-xmt46\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.207004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.206979 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:48.358189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.358155 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg"] Nov 03 21:02:48.362811 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:02:48.362783 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a22bcae_cd8d_4522_bf24_0392d534bfc9.slice/crio-f503f9f3e8f7d8402b9e982df243cf5ae7a8edeb8350f47ac37bc63b254e7818 WatchSource:0}: Error finding container f503f9f3e8f7d8402b9e982df243cf5ae7a8edeb8350f47ac37bc63b254e7818: Status 404 returned error can't find the container with id f503f9f3e8f7d8402b9e982df243cf5ae7a8edeb8350f47ac37bc63b254e7818 Nov 03 21:02:48.592032 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:48.591954 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46"] Nov 03 21:02:48.596635 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:02:48.596591 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44d28a6_142a_49e7_98d6_c1e3b8fca87c.slice/crio-82a59cbb7b07d1f57dfed8c4ad808fd109f7bd2f9bf59d8ee15ce115b829d664 WatchSource:0}: Error finding container 82a59cbb7b07d1f57dfed8c4ad808fd109f7bd2f9bf59d8ee15ce115b829d664: Status 404 returned error can't find the container with id 82a59cbb7b07d1f57dfed8c4ad808fd109f7bd2f9bf59d8ee15ce115b829d664 Nov 03 21:02:49.332822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.332793 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" event={"ID":"6a22bcae-cd8d-4522-bf24-0392d534bfc9","Type":"ContainerStarted","Data":"f503f9f3e8f7d8402b9e982df243cf5ae7a8edeb8350f47ac37bc63b254e7818"} Nov 03 21:02:49.335284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.335252 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" event={"ID":"a44d28a6-142a-49e7-98d6-c1e3b8fca87c","Type":"ContainerStarted","Data":"31de8b45c43eabba33aac1f10b282dabba5020c52ff1fbd42c21984658187a5a"} Nov 03 21:02:49.335410 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.335311 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" event={"ID":"a44d28a6-142a-49e7-98d6-c1e3b8fca87c","Type":"ContainerStarted","Data":"82a59cbb7b07d1f57dfed8c4ad808fd109f7bd2f9bf59d8ee15ce115b829d664"} Nov 03 21:02:49.445294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.445230 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" podStartSLOduration=2.445208744 podStartE2EDuration="2.445208744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:49.443154754 +0000 UTC m=+1977.508396228" watchObservedRunningTime="2026-11-03 21:02:49.445208744 +0000 UTC m=+1977.510450220" Nov 03 21:02:49.609245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.609215 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:49.659206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.659163 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j\" (UniqueName: \"kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j\") pod \"320fcf13-9bc0-412c-acf9-52b8ce758f10\" (UID: \"320fcf13-9bc0-412c-acf9-52b8ce758f10\") " Nov 03 21:02:49.661364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.661336 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j" pod "320fcf13-9bc0-412c-acf9-52b8ce758f10" (UID: "320fcf13-9bc0-412c-acf9-52b8ce758f10"). InnerVolumeSpecName "kube-api-access-tt92j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:49.761149 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:49.761110 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt92j\" (UniqueName: \"kubernetes.io/projected/320fcf13-9bc0-412c-acf9-52b8ce758f10-kube-api-access-tt92j\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:50.054347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.054313 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320fcf13-9bc0-412c-acf9-52b8ce758f10" path="/var/lib/kubelet/pods/320fcf13-9bc0-412c-acf9-52b8ce758f10/volumes" Nov 03 21:02:50.348392 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.348295 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" event={"ID":"6a22bcae-cd8d-4522-bf24-0392d534bfc9","Type":"ContainerStarted","Data":"e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80"} Nov 03 21:02:50.348896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.348692 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:50.351037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.351008 5932 generic.go:356] "Generic (PLEG): container finished" podID="a44d28a6-142a-49e7-98d6-c1e3b8fca87c" containerID="31de8b45c43eabba33aac1f10b282dabba5020c52ff1fbd42c21984658187a5a" exitCode=0 Nov 03 21:02:50.351191 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.351105 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" event={"ID":"a44d28a6-142a-49e7-98d6-c1e3b8fca87c","Type":"ContainerDied","Data":"31de8b45c43eabba33aac1f10b282dabba5020c52ff1fbd42c21984658187a5a"} Nov 03 21:02:50.352679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.352651 5932 generic.go:356] "Generic (PLEG): container finished" podID="320fcf13-9bc0-412c-acf9-52b8ce758f10" containerID="0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0" exitCode=137 Nov 03 21:02:50.352853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.352682 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6927-ff5793" Nov 03 21:02:50.352960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.352690 5932 scope.go:122] "RemoveContainer" containerID="0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0" Nov 03 21:02:50.365185 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.365162 5932 scope.go:122] "RemoveContainer" containerID="0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0" Nov 03 21:02:50.365550 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:02:50.365516 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0\": container with ID starting with 0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0 not found: ID does not exist" containerID="0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0" Nov 03 21:02:50.365666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.365559 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0"} err="failed to get container status \"0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0\": rpc error: code = NotFound desc = could not find container \"0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0\": container with ID starting with 0b55b4fb39abac3f52eab59b4f5e52790fd089627e7fbb69f3ae4746dc10b3e0 not found: ID does not exist" Nov 03 21:02:50.461800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:50.461747 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" podStartSLOduration=3.51338583 podStartE2EDuration="4.461732356s" totalImagesPullingTime="948.346526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:02:50.458724047 +0000 UTC m=+1978.523965519" watchObservedRunningTime="2026-11-03 21:02:50.461732356 +0000 UTC m=+1978.526973826" Nov 03 21:02:51.535234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.535195 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:51.607753 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.607715 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca\") pod \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " Nov 03 21:02:51.607948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.607850 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm\" (UniqueName: \"kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm\") pod \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " Nov 03 21:02:51.608008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.607988 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca\") pod \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\" (UID: \"a44d28a6-142a-49e7-98d6-c1e3b8fca87c\") " Nov 03 21:02:51.608160 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.608136 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca" pod "a44d28a6-142a-49e7-98d6-c1e3b8fca87c" (UID: "a44d28a6-142a-49e7-98d6-c1e3b8fca87c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:02:51.608664 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.608641 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:51.610134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.610111 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca" pod "a44d28a6-142a-49e7-98d6-c1e3b8fca87c" (UID: "a44d28a6-142a-49e7-98d6-c1e3b8fca87c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:02:51.610217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.610168 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm" pod "a44d28a6-142a-49e7-98d6-c1e3b8fca87c" (UID: "a44d28a6-142a-49e7-98d6-c1e3b8fca87c"). InnerVolumeSpecName "kube-api-access-b7dvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:02:51.710150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.710109 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7dvm\" (UniqueName: \"kubernetes.io/projected/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-kube-api-access-b7dvm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:51.710150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:51.710145 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a44d28a6-142a-49e7-98d6-c1e3b8fca87c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:02:52.372591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:52.372490 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" event={"ID":"a44d28a6-142a-49e7-98d6-c1e3b8fca87c","Type":"ContainerDied","Data":"82a59cbb7b07d1f57dfed8c4ad808fd109f7bd2f9bf59d8ee15ce115b829d664"} Nov 03 21:02:52.372591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:52.372550 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a59cbb7b07d1f57dfed8c4ad808fd109f7bd2f9bf59d8ee15ce115b829d664" Nov 03 21:02:52.372919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:52.372604 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46" Nov 03 21:02:56.422408 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:56.421163 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:02:59.389235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:59.389192 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7"] Nov 03 21:02:59.504676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:02:59.504637 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79"] Nov 03 21:03:02.179244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.179212 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn"] Nov 03 21:03:02.180091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.180075 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a44d28a6-142a-49e7-98d6-c1e3b8fca87c" containerName="revoke-keys" Nov 03 21:03:02.180159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.180093 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44d28a6-142a-49e7-98d6-c1e3b8fca87c" containerName="revoke-keys" Nov 03 21:03:02.180241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.180232 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44d28a6-142a-49e7-98d6-c1e3b8fca87c" containerName="revoke-keys" Nov 03 21:03:02.185937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.185916 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.252842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.252801 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn"] Nov 03 21:03:02.281128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.281095 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8448c9-serving-cert\"" Nov 03 21:03:02.312207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.312158 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.312413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.312392 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j5b8\" (UniqueName: \"kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.312466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.312439 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.312600 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.312584 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.413989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.413953 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j5b8\" (UniqueName: \"kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.414158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.414013 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.414158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.414111 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.414238 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.414215 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.414910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.414879 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.416476 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.416453 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.416614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.416590 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.474012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.473932 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j5b8\" (UniqueName: \"kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8\") pod \"maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:02.503860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:02.503826 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:03.002218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:03.002185 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn"] Nov 03 21:03:03.006510 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:03:03.006479 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bce675_6c82_4160_b9c1_8f5a1ecdb419.slice/crio-cab6f81935815673aa5d8317d3e057cbc068b989a5ccba903348248ce190f02f WatchSource:0}: Error finding container cab6f81935815673aa5d8317d3e057cbc068b989a5ccba903348248ce190f02f: Status 404 returned error can't find the container with id cab6f81935815673aa5d8317d3e057cbc068b989a5ccba903348248ce190f02f Nov 03 21:03:03.637981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:03.637938 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" event={"ID":"33bce675-6c82-4160-b9c1-8f5a1ecdb419","Type":"ContainerStarted","Data":"cab6f81935815673aa5d8317d3e057cbc068b989a5ccba903348248ce190f02f"} Nov 03 21:03:04.390546 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:04.390498 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerName="payload-processing" containerID="cri-o://71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec" gracePeriod=25 Nov 03 21:03:04.505299 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:04.505252 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" containerName="payload-pre-processing" containerID="cri-o://9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73" gracePeriod=25 Nov 03 21:03:04.651577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:04.651484 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" event={"ID":"33bce675-6c82-4160-b9c1-8f5a1ecdb419","Type":"ContainerStarted","Data":"c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92"} Nov 03 21:03:04.651577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:04.651556 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:04.765860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:04.765805 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" podStartSLOduration=1.8374638239999999 podStartE2EDuration="2.765791688s" totalImagesPullingTime="928.327864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:04.76279727 +0000 UTC m=+1992.828038744" watchObservedRunningTime="2026-11-03 21:03:04.765791688 +0000 UTC m=+1992.831033161" Nov 03 21:03:06.725142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:06.723732 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": read tcp 10.132.0.2:38552->10.132.0.207:15021: read: connection reset by peer" start-of-body= Nov 03 21:03:06.725142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:06.723796 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": read tcp 10.132.0.2:38552->10.132.0.207:15021: read: connection reset by peer" Nov 03 21:03:07.693177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:07.693145 5932 generic.go:356] "Generic (PLEG): container finished" podID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerID="f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387" exitCode=0 Nov 03 21:03:07.693359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:07.693219 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" event={"ID":"3071cf01-341c-4548-bdad-5b7b4f0b9d41","Type":"ContainerDied","Data":"f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387"} Nov 03 21:03:07.693719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:07.693703 5932 scope.go:122] "RemoveContainer" containerID="f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387" Nov 03 21:03:07.695556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:07.695529 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:03:07.695676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:07.695598 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:03:07.695721 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:07.695659 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:03:08.708110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:08.708074 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" event={"ID":"3071cf01-341c-4548-bdad-5b7b4f0b9d41","Type":"ContainerStarted","Data":"f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b"} Nov 03 21:03:09.700932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.700899 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:03:09.701152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.701090 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:03:09.701235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.701142 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 21:03:09.949404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.949365 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz"] Nov 03 21:03:09.953408 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.953349 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:09.966996 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.966971 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:09.967138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.967034 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:09.967138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.967076 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fx9r\" (UniqueName: \"kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:09.997994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:09.997959 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz"] Nov 03 21:03:10.045975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.045934 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:03:10.049861 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.049609 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:10.068992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.068949 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.069194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.069049 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.069194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.069075 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fx9r\" (UniqueName: \"kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.069799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.069775 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.071425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.071402 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.128066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.127999 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fx9r\" (UniqueName: \"kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.268378 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.268291 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:10.668611 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.668576 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:03:10.674335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.674305 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz"] Nov 03 21:03:10.678495 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:03:10.678464 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf8266f_0fdd_41d7_8e54_20eacf383629.slice/crio-7376daae9fd5d5e974efad84b97f837c569579673dbb105b0d2f3cf907d98d9b WatchSource:0}: Error finding container 7376daae9fd5d5e974efad84b97f837c569579673dbb105b0d2f3cf907d98d9b: Status 404 returned error can't find the container with id 7376daae9fd5d5e974efad84b97f837c569579673dbb105b0d2f3cf907d98d9b Nov 03 21:03:10.701735 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.701708 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:03:10.701852 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.701759 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 21:03:10.760178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:10.760139 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" event={"ID":"2bf8266f-0fdd-41d7-8e54-20eacf383629","Type":"ContainerStarted","Data":"7376daae9fd5d5e974efad84b97f837c569579673dbb105b0d2f3cf907d98d9b"} Nov 03 21:03:11.364852 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.364756 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84"] Nov 03 21:03:11.370180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.370159 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.410709 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.410667 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.410920 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.410840 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfx7k\" (UniqueName: \"kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.410920 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.410908 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.417035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.417004 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84"] Nov 03 21:03:11.512001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.511962 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfx7k\" (UniqueName: \"kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.512209 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.512024 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.512289 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.512209 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.513187 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.513164 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.515727 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.515699 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.571506 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.571469 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfx7k\" (UniqueName: \"kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k\") pod \"maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.687131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.687101 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:11.700469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.700440 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:03:11.700586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.700491 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 21:03:11.777278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:11.777219 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" event={"ID":"2bf8266f-0fdd-41d7-8e54-20eacf383629","Type":"ContainerStarted","Data":"4c5a02fd62c07a95e3071427e552f854648326643dbe00a797b46bf33a589f1a"} Nov 03 21:03:12.097857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.097789 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" podStartSLOduration=3.097771115 podStartE2EDuration="3.097771115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:11.905602238 +0000 UTC m=+1999.970843710" watchObservedRunningTime="2026-11-03 21:03:12.097771115 +0000 UTC m=+2000.163012587" Nov 03 21:03:12.099591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.099556 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84"] Nov 03 21:03:12.105297 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:03:12.105260 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54f9c4a_b1f4_488c_a3a6_b767d5d8393c.slice/crio-5302c3ca4f10b4b756974cc166f3d968cd09777bbcdc83620858f892782cea34 WatchSource:0}: Error finding container 5302c3ca4f10b4b756974cc166f3d968cd09777bbcdc83620858f892782cea34: Status 404 returned error can't find the container with id 5302c3ca4f10b4b756974cc166f3d968cd09777bbcdc83620858f892782cea34 Nov 03 21:03:12.445647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.439515 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:12.445647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.439818 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:12.451067 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:12.451038 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54f9c4a_b1f4_488c_a3a6_b767d5d8393c.slice/crio-4657a01f6cfba84146e497a84b1ec111215d3772984b429659eeb89c6d769c60.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:03:12.700654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.700553 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:03:12.700801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.700614 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 21:03:12.789741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.789702 5932 generic.go:356] "Generic (PLEG): container finished" podID="2bf8266f-0fdd-41d7-8e54-20eacf383629" containerID="4c5a02fd62c07a95e3071427e552f854648326643dbe00a797b46bf33a589f1a" exitCode=0 Nov 03 21:03:12.789932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.789780 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" event={"ID":"2bf8266f-0fdd-41d7-8e54-20eacf383629","Type":"ContainerDied","Data":"4c5a02fd62c07a95e3071427e552f854648326643dbe00a797b46bf33a589f1a"} Nov 03 21:03:12.791465 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.791437 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" event={"ID":"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c","Type":"ContainerStarted","Data":"4657a01f6cfba84146e497a84b1ec111215d3772984b429659eeb89c6d769c60"} Nov 03 21:03:12.791574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:12.791486 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" event={"ID":"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c","Type":"ContainerStarted","Data":"5302c3ca4f10b4b756974cc166f3d968cd09777bbcdc83620858f892782cea34"} Nov 03 21:03:13.051768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:13.051647 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" podStartSLOduration=2.05161367 podStartE2EDuration="2.05161367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:13.04648376 +0000 UTC m=+2001.111725235" watchObservedRunningTime="2026-11-03 21:03:13.05161367 +0000 UTC m=+2001.116855142" Nov 03 21:03:13.701236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:13.701202 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Nov 03 21:03:13.701658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:13.701266 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Nov 03 21:03:13.801567 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:13.801526 5932 generic.go:356] "Generic (PLEG): container finished" podID="e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" containerID="4657a01f6cfba84146e497a84b1ec111215d3772984b429659eeb89c6d769c60" exitCode=0 Nov 03 21:03:13.801764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:13.801603 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" event={"ID":"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c","Type":"ContainerDied","Data":"4657a01f6cfba84146e497a84b1ec111215d3772984b429659eeb89c6d769c60"} Nov 03 21:03:14.000814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.000787 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:14.075166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.075126 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r\" (UniqueName: \"kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r\") pod \"2bf8266f-0fdd-41d7-8e54-20eacf383629\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " Nov 03 21:03:14.075370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.075176 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca\") pod \"2bf8266f-0fdd-41d7-8e54-20eacf383629\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " Nov 03 21:03:14.075370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.075299 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca\") pod \"2bf8266f-0fdd-41d7-8e54-20eacf383629\" (UID: \"2bf8266f-0fdd-41d7-8e54-20eacf383629\") " Nov 03 21:03:14.075589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.075561 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca" pod "2bf8266f-0fdd-41d7-8e54-20eacf383629" (UID: "2bf8266f-0fdd-41d7-8e54-20eacf383629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:14.075902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.075882 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:14.077357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.077335 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca" pod "2bf8266f-0fdd-41d7-8e54-20eacf383629" (UID: "2bf8266f-0fdd-41d7-8e54-20eacf383629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:14.077506 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.077485 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r" pod "2bf8266f-0fdd-41d7-8e54-20eacf383629" (UID: "2bf8266f-0fdd-41d7-8e54-20eacf383629"). InnerVolumeSpecName "kube-api-access-7fx9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:14.164922 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:14.164877 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:03:14.177128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.177098 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bf8266f-0fdd-41d7-8e54-20eacf383629-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:14.177128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.177125 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fx9r\" (UniqueName: \"kubernetes.io/projected/2bf8266f-0fdd-41d7-8e54-20eacf383629-kube-api-access-7fx9r\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:14.705760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.705730 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:03:14.706262 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.706242 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:03:14.707064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.707043 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:03:14.813616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.813571 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" event={"ID":"2bf8266f-0fdd-41d7-8e54-20eacf383629","Type":"ContainerDied","Data":"7376daae9fd5d5e974efad84b97f837c569579673dbb105b0d2f3cf907d98d9b"} Nov 03 21:03:14.813826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.813589 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz" Nov 03 21:03:14.813826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.813690 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7376daae9fd5d5e974efad84b97f837c569579673dbb105b0d2f3cf907d98d9b" Nov 03 21:03:14.985910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:14.985877 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:15.103364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.103327 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca\") pod \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " Nov 03 21:03:15.103695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.103427 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k\" (UniqueName: \"kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k\") pod \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " Nov 03 21:03:15.103832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.103804 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca\") pod \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\" (UID: \"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c\") " Nov 03 21:03:15.117331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.110888 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k" pod "e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" (UID: "e54f9c4a-b1f4-488c-a3a6-b767d5d8393c"). InnerVolumeSpecName "kube-api-access-zfx7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:15.117331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.116821 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca" pod "e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" (UID: "e54f9c4a-b1f4-488c-a3a6-b767d5d8393c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:15.118227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.117526 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:15.118227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.117555 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfx7k\" (UniqueName: \"kubernetes.io/projected/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-kube-api-access-zfx7k\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:15.119397 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.119369 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca" pod "e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" (UID: "e54f9c4a-b1f4-488c-a3a6-b767d5d8393c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:15.219048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.218964 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:15.824543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.824507 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" event={"ID":"e54f9c4a-b1f4-488c-a3a6-b767d5d8393c","Type":"ContainerDied","Data":"5302c3ca4f10b4b756974cc166f3d968cd09777bbcdc83620858f892782cea34"} Nov 03 21:03:15.824543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.824533 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84" Nov 03 21:03:15.825062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:15.824569 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5302c3ca4f10b4b756974cc166f3d968cd09777bbcdc83620858f892782cea34" Nov 03 21:03:21.178761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:21.178719 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc"] Nov 03 21:03:22.424891 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:22.424850 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:22.425306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:22.424850 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:03:23.585833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:23.585803 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz"] Nov 03 21:03:23.700636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:23.700584 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn"] Nov 03 21:03:24.717854 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.717821 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 21:03:24.794088 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.794047 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume\") pod \"1f949d01-9ca7-4055-a159-91bb637bc3d3\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " Nov 03 21:03:24.794282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.794244 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg\" (UniqueName: \"kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg\") pod \"1f949d01-9ca7-4055-a159-91bb637bc3d3\" (UID: \"1f949d01-9ca7-4055-a159-91bb637bc3d3\") " Nov 03 21:03:24.794555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.794524 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume" pod "1f949d01-9ca7-4055-a159-91bb637bc3d3" (UID: "1f949d01-9ca7-4055-a159-91bb637bc3d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:24.794977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.794955 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f949d01-9ca7-4055-a159-91bb637bc3d3-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:24.797236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.797209 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg" pod "1f949d01-9ca7-4055-a159-91bb637bc3d3" (UID: "1f949d01-9ca7-4055-a159-91bb637bc3d3"). InnerVolumeSpecName "kube-api-access-qk6lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:24.831874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.831159 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 21:03:24.895423 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.895391 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume\") pod \"41e72204-f86e-4db8-a1cf-5677653359ef\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " Nov 03 21:03:24.895597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.895469 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft\" (UniqueName: \"kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft\") pod \"41e72204-f86e-4db8-a1cf-5677653359ef\" (UID: \"41e72204-f86e-4db8-a1cf-5677653359ef\") " Nov 03 21:03:24.895868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.895845 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume" pod "41e72204-f86e-4db8-a1cf-5677653359ef" (UID: "41e72204-f86e-4db8-a1cf-5677653359ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:24.896204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.896186 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41e72204-f86e-4db8-a1cf-5677653359ef-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:24.896253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.896207 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6lg\" (UniqueName: \"kubernetes.io/projected/1f949d01-9ca7-4055-a159-91bb637bc3d3-kube-api-access-qk6lg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:24.897589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.897570 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft" pod "41e72204-f86e-4db8-a1cf-5677653359ef" (UID: "41e72204-f86e-4db8-a1cf-5677653359ef"). InnerVolumeSpecName "kube-api-access-vggft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:24.963710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.963587 5932 generic.go:356] "Generic (PLEG): container finished" podID="41e72204-f86e-4db8-a1cf-5677653359ef" containerID="9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73" exitCode=0 Nov 03 21:03:24.963710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.963692 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" Nov 03 21:03:24.963710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.963690 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" event={"ID":"41e72204-f86e-4db8-a1cf-5677653359ef","Type":"ContainerDied","Data":"9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73"} Nov 03 21:03:24.963999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.963792 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79" event={"ID":"41e72204-f86e-4db8-a1cf-5677653359ef","Type":"ContainerDied","Data":"0e188a8c64240035d83116e041b3c4edd8b7acf1bb95069b539201e458d00da2"} Nov 03 21:03:24.963999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.963830 5932 scope.go:122] "RemoveContainer" containerID="9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73" Nov 03 21:03:24.965955 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.965928 5932 generic.go:356] "Generic (PLEG): container finished" podID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerID="71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec" exitCode=0 Nov 03 21:03:24.966098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.966002 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" event={"ID":"1f949d01-9ca7-4055-a159-91bb637bc3d3","Type":"ContainerDied","Data":"71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec"} Nov 03 21:03:24.966098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.966040 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" event={"ID":"1f949d01-9ca7-4055-a159-91bb637bc3d3","Type":"ContainerDied","Data":"d235fe585dd5c436d47e2b3efb93eb7a69514270fdb818f622004258b02e34bd"} Nov 03 21:03:24.966218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.966118 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7" Nov 03 21:03:24.978518 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.978494 5932 scope.go:122] "RemoveContainer" containerID="9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73" Nov 03 21:03:24.978837 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:24.978815 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73\": container with ID starting with 9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73 not found: ID does not exist" containerID="9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73" Nov 03 21:03:24.978938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.978843 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73"} err="failed to get container status \"9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73\": rpc error: code = NotFound desc = could not find container \"9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73\": container with ID starting with 9c7def838f99f96cdc8dfcfefbc362afdd7e7a40ce1e8fa41ece10843ded2f73 not found: ID does not exist" Nov 03 21:03:24.978938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.978861 5932 scope.go:122] "RemoveContainer" containerID="71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec" Nov 03 21:03:24.992086 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.992059 5932 scope.go:122] "RemoveContainer" containerID="71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec" Nov 03 21:03:24.992368 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:24.992349 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec\": container with ID starting with 71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec not found: ID does not exist" containerID="71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec" Nov 03 21:03:24.992415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.992376 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec"} err="failed to get container status \"71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec\": rpc error: code = NotFound desc = could not find container \"71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec\": container with ID starting with 71bff2ddac77aaf778be0564750f649b3b4a37a08a52ce0acc42bac99a6ea9ec not found: ID does not exist" Nov 03 21:03:24.997397 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:24.997379 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vggft\" (UniqueName: \"kubernetes.io/projected/41e72204-f86e-4db8-a1cf-5677653359ef-kube-api-access-vggft\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:25.161648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:25.160668 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79"] Nov 03 21:03:25.217743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:25.216868 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e68f8112-8574869b79-r2n79"] Nov 03 21:03:25.295646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:25.295155 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7"] Nov 03 21:03:25.295646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:25.295223 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e68f8112-6874f8477c-m76s7"] Nov 03 21:03:26.053997 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.053963 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" path="/var/lib/kubelet/pods/1f949d01-9ca7-4055-a159-91bb637bc3d3/volumes" Nov 03 21:03:26.054442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.054428 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" path="/var/lib/kubelet/pods/41e72204-f86e-4db8-a1cf-5677653359ef/volumes" Nov 03 21:03:26.179426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.179382 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" podUID="9e8a3ea4-6468-482e-87c3-c82190f5717b" containerName="payload-pre-processing" containerID="cri-o://1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8" gracePeriod=25 Nov 03 21:03:26.465217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.465184 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 21:03:26.646646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.646255 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp\" (UniqueName: \"kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp\") pod \"9e8a3ea4-6468-482e-87c3-c82190f5717b\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " Nov 03 21:03:26.646646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.646357 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume\") pod \"9e8a3ea4-6468-482e-87c3-c82190f5717b\" (UID: \"9e8a3ea4-6468-482e-87c3-c82190f5717b\") " Nov 03 21:03:26.692215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.692153 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp" pod "9e8a3ea4-6468-482e-87c3-c82190f5717b" (UID: "9e8a3ea4-6468-482e-87c3-c82190f5717b"). InnerVolumeSpecName "kube-api-access-8x7jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:26.699655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.694918 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume" pod "9e8a3ea4-6468-482e-87c3-c82190f5717b" (UID: "9e8a3ea4-6468-482e-87c3-c82190f5717b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:26.749987 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.749948 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7jp\" (UniqueName: \"kubernetes.io/projected/9e8a3ea4-6468-482e-87c3-c82190f5717b-kube-api-access-8x7jp\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:26.750232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.750215 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8a3ea4-6468-482e-87c3-c82190f5717b-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:26.987091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.986906 5932 generic.go:356] "Generic (PLEG): container finished" podID="9e8a3ea4-6468-482e-87c3-c82190f5717b" containerID="1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8" exitCode=0 Nov 03 21:03:26.987091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.986968 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" Nov 03 21:03:26.987091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.986971 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" event={"ID":"9e8a3ea4-6468-482e-87c3-c82190f5717b","Type":"ContainerDied","Data":"1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8"} Nov 03 21:03:26.987383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.987102 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc" event={"ID":"9e8a3ea4-6468-482e-87c3-c82190f5717b","Type":"ContainerDied","Data":"cfa7012cfa61e94062352602fd56ed314a0e7c13b4a7938fbb5d9e3e9cdd2d65"} Nov 03 21:03:26.987383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.987132 5932 scope.go:122] "RemoveContainer" containerID="1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8" Nov 03 21:03:26.999266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.999244 5932 scope.go:122] "RemoveContainer" containerID="1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8" Nov 03 21:03:26.999604 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:26.999581 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8\": container with ID starting with 1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8 not found: ID does not exist" containerID="1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8" Nov 03 21:03:26.999705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:26.999615 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8"} err="failed to get container status \"1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8\": rpc error: code = NotFound desc = could not find container \"1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8\": container with ID starting with 1dfef646eb8cc561d381d79e194a05c056bbb1fc859521753847cfe0620bc2f8 not found: ID does not exist" Nov 03 21:03:27.161731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:27.161698 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc"] Nov 03 21:03:27.207660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:27.207583 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-5b12d6-c7f5546f5-m9qlc"] Nov 03 21:03:28.161645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:28.149640 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e8a3ea4-6468-482e-87c3-c82190f5717b" path="/var/lib/kubelet/pods/9e8a3ea4-6468-482e-87c3-c82190f5717b/volumes" Nov 03 21:03:28.587070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:28.586968 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" podUID="b1c572b5-3c9b-4272-ac29-fbb79385cc85" containerName="payload-processing" containerID="cri-o://f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045" gracePeriod=25 Nov 03 21:03:28.701904 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:28.701857 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" podUID="04832731-fd34-4c3a-aa4d-ea769ad586bd" containerName="payload-pre-processing" containerID="cri-o://316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8" gracePeriod=25 Nov 03 21:03:28.896222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:28.896193 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:03:29.002038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.002007 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:03:29.004305 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.004277 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume\") pod \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " Nov 03 21:03:29.004456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.004337 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf\" (UniqueName: \"kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf\") pod \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\" (UID: \"b1c572b5-3c9b-4272-ac29-fbb79385cc85\") " Nov 03 21:03:29.004737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.004704 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume" pod "b1c572b5-3c9b-4272-ac29-fbb79385cc85" (UID: "b1c572b5-3c9b-4272-ac29-fbb79385cc85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:29.005266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.005241 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1c572b5-3c9b-4272-ac29-fbb79385cc85-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:29.008703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.008634 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf" pod "b1c572b5-3c9b-4272-ac29-fbb79385cc85" (UID: "b1c572b5-3c9b-4272-ac29-fbb79385cc85"). InnerVolumeSpecName "kube-api-access-t5mgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:29.014034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.014003 5932 generic.go:356] "Generic (PLEG): container finished" podID="b1c572b5-3c9b-4272-ac29-fbb79385cc85" containerID="f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045" exitCode=0 Nov 03 21:03:29.014129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.014051 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" Nov 03 21:03:29.014129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.014083 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" event={"ID":"b1c572b5-3c9b-4272-ac29-fbb79385cc85","Type":"ContainerDied","Data":"f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045"} Nov 03 21:03:29.014241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.014162 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz" event={"ID":"b1c572b5-3c9b-4272-ac29-fbb79385cc85","Type":"ContainerDied","Data":"9be14dc3c7d115333980474770b1a1116b184d59b6fcb2421c871982e14a969e"} Nov 03 21:03:29.014241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.014191 5932 scope.go:122] "RemoveContainer" containerID="f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045" Nov 03 21:03:29.016020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.015995 5932 generic.go:356] "Generic (PLEG): container finished" podID="04832731-fd34-4c3a-aa4d-ea769ad586bd" containerID="316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8" exitCode=0 Nov 03 21:03:29.016144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.016050 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" Nov 03 21:03:29.016240 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.016211 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" event={"ID":"04832731-fd34-4c3a-aa4d-ea769ad586bd","Type":"ContainerDied","Data":"316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8"} Nov 03 21:03:29.016314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.016263 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn" event={"ID":"04832731-fd34-4c3a-aa4d-ea769ad586bd","Type":"ContainerDied","Data":"db4a82406d59455d72db9892c84bbc3ce985f2ce2ecb1a9870b5bf78a1bbcbde"} Nov 03 21:03:29.028778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.028688 5932 scope.go:122] "RemoveContainer" containerID="f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045" Nov 03 21:03:29.029043 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:29.029025 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045\": container with ID starting with f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045 not found: ID does not exist" containerID="f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045" Nov 03 21:03:29.029108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.029054 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045"} err="failed to get container status \"f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045\": rpc error: code = NotFound desc = could not find container \"f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045\": container with ID starting with f751af685332edfe804d61d6bd36ecbfda59bad9c55f3d5e4b5716dd7dccb045 not found: ID does not exist" Nov 03 21:03:29.029108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.029071 5932 scope.go:122] "RemoveContainer" containerID="316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8" Nov 03 21:03:29.039658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.039605 5932 scope.go:122] "RemoveContainer" containerID="316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8" Nov 03 21:03:29.039984 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:29.039962 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8\": container with ID starting with 316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8 not found: ID does not exist" containerID="316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8" Nov 03 21:03:29.040049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.039992 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8"} err="failed to get container status \"316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8\": rpc error: code = NotFound desc = could not find container \"316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8\": container with ID starting with 316a4fe61ad64c646ae0b47cce23e49d6adbf242cb072122d09a7de8d252bfc8 not found: ID does not exist" Nov 03 21:03:29.106469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.106384 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v\" (UniqueName: \"kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v\") pod \"04832731-fd34-4c3a-aa4d-ea769ad586bd\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " Nov 03 21:03:29.106632 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.106558 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume\") pod \"04832731-fd34-4c3a-aa4d-ea769ad586bd\" (UID: \"04832731-fd34-4c3a-aa4d-ea769ad586bd\") " Nov 03 21:03:29.106977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.106950 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume" pod "04832731-fd34-4c3a-aa4d-ea769ad586bd" (UID: "04832731-fd34-4c3a-aa4d-ea769ad586bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:29.107352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.107332 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04832731-fd34-4c3a-aa4d-ea769ad586bd-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:29.107399 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.107357 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mgf\" (UniqueName: \"kubernetes.io/projected/b1c572b5-3c9b-4272-ac29-fbb79385cc85-kube-api-access-t5mgf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:29.108657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.108614 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v" pod "04832731-fd34-4c3a-aa4d-ea769ad586bd" (UID: "04832731-fd34-4c3a-aa4d-ea769ad586bd"). InnerVolumeSpecName "kube-api-access-cfb2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:29.208605 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.208564 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfb2v\" (UniqueName: \"kubernetes.io/projected/04832731-fd34-4c3a-aa4d-ea769ad586bd-kube-api-access-cfb2v\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:29.285730 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.285685 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz"] Nov 03 21:03:29.290067 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.290042 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-af2e8a-58b6c7b54f-jqpkz"] Nov 03 21:03:29.431105 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.431071 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn"] Nov 03 21:03:29.436868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:29.436837 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-af2e8a-557f9fdf7f-zd5hn"] Nov 03 21:03:30.054220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:30.054186 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04832731-fd34-4c3a-aa4d-ea769ad586bd" path="/var/lib/kubelet/pods/04832731-fd34-4c3a-aa4d-ea769ad586bd/volumes" Nov 03 21:03:30.054705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:30.054690 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c572b5-3c9b-4272-ac29-fbb79385cc85" path="/var/lib/kubelet/pods/b1c572b5-3c9b-4272-ac29-fbb79385cc85/volumes" Nov 03 21:03:39.992013 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.991977 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv"] Nov 03 21:03:39.992791 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992775 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2bf8266f-0fdd-41d7-8e54-20eacf383629" containerName="revoke-keys" Nov 03 21:03:39.992859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992792 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf8266f-0fdd-41d7-8e54-20eacf383629" containerName="revoke-keys" Nov 03 21:03:39.992859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992812 5932 cpu_manager.go:414] "containerMap: removing container" podUID="04832731-fd34-4c3a-aa4d-ea769ad586bd" containerName="payload-pre-processing" Nov 03 21:03:39.992859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992817 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04832731-fd34-4c3a-aa4d-ea769ad586bd" containerName="payload-pre-processing" Nov 03 21:03:39.992859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992842 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" containerName="revoke-keys" Nov 03 21:03:39.992859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992847 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" containerName="revoke-keys" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992868 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b1c572b5-3c9b-4272-ac29-fbb79385cc85" containerName="payload-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992873 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c572b5-3c9b-4272-ac29-fbb79385cc85" containerName="payload-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992880 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerName="payload-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992885 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerName="payload-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992897 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9e8a3ea4-6468-482e-87c3-c82190f5717b" containerName="payload-pre-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992902 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e8a3ea4-6468-482e-87c3-c82190f5717b" containerName="payload-pre-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992910 5932 cpu_manager.go:414] "containerMap: removing container" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" containerName="payload-pre-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.992914 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" containerName="payload-pre-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993021 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e8a3ea4-6468-482e-87c3-c82190f5717b" containerName="payload-pre-processing" Nov 03 21:03:39.993036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993030 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="41e72204-f86e-4db8-a1cf-5677653359ef" containerName="payload-pre-processing" Nov 03 21:03:39.993339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993040 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf8266f-0fdd-41d7-8e54-20eacf383629" containerName="revoke-keys" Nov 03 21:03:39.993339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993050 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="04832731-fd34-4c3a-aa4d-ea769ad586bd" containerName="payload-pre-processing" Nov 03 21:03:39.993339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993060 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" containerName="revoke-keys" Nov 03 21:03:39.993339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993070 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c572b5-3c9b-4272-ac29-fbb79385cc85" containerName="payload-processing" Nov 03 21:03:39.993339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.993077 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f949d01-9ca7-4055-a159-91bb637bc3d3" containerName="payload-processing" Nov 03 21:03:39.998424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:39.998400 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.055893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.055861 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv"] Nov 03 21:03:40.069734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.069692 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.069908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.069741 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd4dx\" (UniqueName: \"kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.069908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.069888 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.070066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.069955 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.092705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.092664 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-878695-serving-cert\"" Nov 03 21:03:40.171784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.171748 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.171966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.171797 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd4dx\" (UniqueName: \"kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.171966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.171881 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.171966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.171948 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.172106 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:40.172084 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-serving-cert: secret "maas-api-e2e-inf-a-878695-serving-cert" not found Nov 03 21:03:40.172198 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:40.172186 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls podName:adf97a22-3157-4516-9cdf-de395e9b0faf nodeName:}" failed. No retries permitted until 2026-11-03 21:03:40.67216283 +0000 UTC m=+2028.737404298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls") pod "maas-api-e2e-inf-a-878695-6b7458d845-2flmv" (UID: "adf97a22-3157-4516-9cdf-de395e9b0faf") : secret "maas-api-e2e-inf-a-878695-serving-cert" not found Nov 03 21:03:40.172532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.172508 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.174274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.174251 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.229747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.229716 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd4dx\" (UniqueName: \"kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.680762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.680724 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.683247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.683220 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\") pod \"maas-api-e2e-inf-a-878695-6b7458d845-2flmv\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:40.914483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:40.914451 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:41.293707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:41.293602 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv"] Nov 03 21:03:41.298133 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:03:41.298107 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf97a22_3157_4516_9cdf_de395e9b0faf.slice/crio-89c2567053ab8b4326f96d6b0fd082a3925d4d47c8f4af4648ae949f6b43721e WatchSource:0}: Error finding container 89c2567053ab8b4326f96d6b0fd082a3925d4d47c8f4af4648ae949f6b43721e: Status 404 returned error can't find the container with id 89c2567053ab8b4326f96d6b0fd082a3925d4d47c8f4af4648ae949f6b43721e Nov 03 21:03:42.147031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:42.146994 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" event={"ID":"adf97a22-3157-4516-9cdf-de395e9b0faf","Type":"ContainerStarted","Data":"89c2567053ab8b4326f96d6b0fd082a3925d4d47c8f4af4648ae949f6b43721e"} Nov 03 21:03:43.158663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:43.158592 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" event={"ID":"adf97a22-3157-4516-9cdf-de395e9b0faf","Type":"ContainerStarted","Data":"f542f8d35912652384ad272401e626ff2a3c75de7e7f19a724b4eafc9a0cc599"} Nov 03 21:03:43.159250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:43.158710 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:43.271667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:43.271603 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" podStartSLOduration=3.351143552 podStartE2EDuration="4.271589603s" totalImagesPullingTime="920.446051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:03:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:03:43.26797349 +0000 UTC m=+2031.333214963" watchObservedRunningTime="2026-11-03 21:03:43.271589603 +0000 UTC m=+2031.336831075" Nov 03 21:03:44.152086 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:44.152047 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:03:44.228162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.228120 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg"] Nov 03 21:03:44.228680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.228372 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" containerID="cri-o://d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164" gracePeriod=30 Nov 03 21:03:44.397351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.397318 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn"] Nov 03 21:03:44.402693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.402633 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.457470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.457432 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn"] Nov 03 21:03:44.468394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.468348 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:03:44.469091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.468758 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:44.551595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.551563 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.551595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.551602 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rctht\" (UniqueName: \"kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.551943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.551916 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.653308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.653232 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.653308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.653298 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.653552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.653322 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctht\" (UniqueName: \"kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.653939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.653917 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.655760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.655739 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.711453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.711419 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctht\" (UniqueName: \"kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht\") pod \"maas-api-revoke-sub-8faf882d-4w9bn\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:44.718387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:44.718362 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:45.120480 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:45.120403 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn"] Nov 03 21:03:45.124923 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:03:45.124896 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19154b12_c9d0_418c_8963_35a9ee86efb2.slice/crio-23b1f0ae4f86cf55bc40756dc5f9994702f51c8382cf12bf22d27e34975445a8 WatchSource:0}: Error finding container 23b1f0ae4f86cf55bc40756dc5f9994702f51c8382cf12bf22d27e34975445a8: Status 404 returned error can't find the container with id 23b1f0ae4f86cf55bc40756dc5f9994702f51c8382cf12bf22d27e34975445a8 Nov 03 21:03:45.177488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:45.177451 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" event={"ID":"19154b12-c9d0-418c-8963-35a9ee86efb2","Type":"ContainerStarted","Data":"23b1f0ae4f86cf55bc40756dc5f9994702f51c8382cf12bf22d27e34975445a8"} Nov 03 21:03:46.188557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:46.188523 5932 generic.go:356] "Generic (PLEG): container finished" podID="19154b12-c9d0-418c-8963-35a9ee86efb2" containerID="3851d56ebeff1f2184eaec8ab3e3c64e9b1f016db85a2608157864ed4c0ce4b8" exitCode=0 Nov 03 21:03:46.188971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:46.188595 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" event={"ID":"19154b12-c9d0-418c-8963-35a9ee86efb2","Type":"ContainerDied","Data":"3851d56ebeff1f2184eaec8ab3e3c64e9b1f016db85a2608157864ed4c0ce4b8"} Nov 03 21:03:47.360636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.360606 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:47.396777 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.396751 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca\") pod \"19154b12-c9d0-418c-8963-35a9ee86efb2\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " Nov 03 21:03:47.396929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.396810 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca\") pod \"19154b12-c9d0-418c-8963-35a9ee86efb2\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " Nov 03 21:03:47.396979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.396943 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht\" (UniqueName: \"kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht\") pod \"19154b12-c9d0-418c-8963-35a9ee86efb2\" (UID: \"19154b12-c9d0-418c-8963-35a9ee86efb2\") " Nov 03 21:03:47.397101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.397080 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca" pod "19154b12-c9d0-418c-8963-35a9ee86efb2" (UID: "19154b12-c9d0-418c-8963-35a9ee86efb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:47.397405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.397378 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.398880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.398855 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca" pod "19154b12-c9d0-418c-8963-35a9ee86efb2" (UID: "19154b12-c9d0-418c-8963-35a9ee86efb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:47.399029 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.399011 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht" pod "19154b12-c9d0-418c-8963-35a9ee86efb2" (UID: "19154b12-c9d0-418c-8963-35a9ee86efb2"). InnerVolumeSpecName "kube-api-access-rctht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:47.499490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.499412 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rctht\" (UniqueName: \"kubernetes.io/projected/19154b12-c9d0-418c-8963-35a9ee86efb2-kube-api-access-rctht\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:47.499490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:47.499453 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19154b12-c9d0-418c-8963-35a9ee86efb2-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:48.207195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:48.207161 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" Nov 03 21:03:48.207377 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:48.207191 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn" event={"ID":"19154b12-c9d0-418c-8963-35a9ee86efb2","Type":"ContainerDied","Data":"23b1f0ae4f86cf55bc40756dc5f9994702f51c8382cf12bf22d27e34975445a8"} Nov 03 21:03:48.207377 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:48.207241 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23b1f0ae4f86cf55bc40756dc5f9994702f51c8382cf12bf22d27e34975445a8" Nov 03 21:03:49.172821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.172788 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:03:49.538809 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.538783 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 21:03:49.635337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635305 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.635561 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635357 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.635561 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635390 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.635561 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635453 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.635561 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635517 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.635853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635680 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.635853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635834 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:49.635964 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635851 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:49.636019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.635985 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:49.636080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.636014 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:49.636133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.636086 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.636184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.636129 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.636253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.636235 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo\") pod \"0087567b-0be5-4ce8-816c-1ce888df102b\" (UID: \"0087567b-0be5-4ce8-816c-1ce888df102b\") " Nov 03 21:03:49.636425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.636399 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:49.637049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637023 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0087567b-0be5-4ce8-816c-1ce888df102b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.637152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637052 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.637152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637066 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.637152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637079 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.637152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637092 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.637866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637846 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "kube-api-access-hbzv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:49.637956 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.637873 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:49.638209 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.638185 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:03:49.638273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.638254 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo" pod "0087567b-0be5-4ce8-816c-1ce888df102b" (UID: "0087567b-0be5-4ce8-816c-1ce888df102b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:03:49.739125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.739035 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0087567b-0be5-4ce8-816c-1ce888df102b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.739125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.739074 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0087567b-0be5-4ce8-816c-1ce888df102b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.739125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.739089 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzv4\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-kube-api-access-hbzv4\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:49.739125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:49.739106 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0087567b-0be5-4ce8-816c-1ce888df102b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:50.225197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.225158 5932 generic.go:356] "Generic (PLEG): container finished" podID="0087567b-0be5-4ce8-816c-1ce888df102b" containerID="d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164" exitCode=0 Nov 03 21:03:50.225669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.225218 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" Nov 03 21:03:50.225669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.225243 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" event={"ID":"0087567b-0be5-4ce8-816c-1ce888df102b","Type":"ContainerDied","Data":"d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164"} Nov 03 21:03:50.225669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.225302 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg" event={"ID":"0087567b-0be5-4ce8-816c-1ce888df102b","Type":"ContainerDied","Data":"56cdcbf6b2fbbe481b8dec84ec0377dfe682a1873c5c5cca50afeeda1086429f"} Nov 03 21:03:50.225669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.225326 5932 scope.go:122] "RemoveContainer" containerID="d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164" Nov 03 21:03:50.236897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.236873 5932 scope.go:122] "RemoveContainer" containerID="d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164" Nov 03 21:03:50.237184 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:03:50.237154 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164\": container with ID starting with d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164 not found: ID does not exist" containerID="d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164" Nov 03 21:03:50.237253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.237198 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164"} err="failed to get container status \"d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164\": rpc error: code = NotFound desc = could not find container \"d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164\": container with ID starting with d92fe425a7784d09612735d0a9556b086d58b4ac13032dfd365c66a5ff28f164 not found: ID does not exist" Nov 03 21:03:50.384784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.384752 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg"] Nov 03 21:03:50.388789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:50.388768 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-4bggg"] Nov 03 21:03:51.421229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.421198 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25"] Nov 03 21:03:51.422012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.421997 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" Nov 03 21:03:51.422103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.422013 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" Nov 03 21:03:51.422103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.422026 5932 cpu_manager.go:414] "containerMap: removing container" podUID="19154b12-c9d0-418c-8963-35a9ee86efb2" containerName="revoke-keys" Nov 03 21:03:51.422103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.422032 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19154b12-c9d0-418c-8963-35a9ee86efb2" containerName="revoke-keys" Nov 03 21:03:51.422206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.422161 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" containerName="istio-proxy" Nov 03 21:03:51.422206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.422173 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="19154b12-c9d0-418c-8963-35a9ee86efb2" containerName="revoke-keys" Nov 03 21:03:51.427497 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.427480 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.462589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.462558 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.462767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.462604 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rf9\" (UniqueName: \"kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.462767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.462696 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.481865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.481835 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25"] Nov 03 21:03:51.521171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.521139 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:03:51.521353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.521218 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:03:51.564292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.564261 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.564438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.564300 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rf9\" (UniqueName: \"kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.564438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.564334 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.565087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.565064 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.566746 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.566726 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.621914 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.621888 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rf9\" (UniqueName: \"kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9\") pod \"maas-api-revoke-keys-e2e-api-a-ba7e83-wks25\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:51.743833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:51.743764 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:52.055618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:52.055544 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0087567b-0be5-4ce8-816c-1ce888df102b" path="/var/lib/kubelet/pods/0087567b-0be5-4ce8-816c-1ce888df102b/volumes" Nov 03 21:03:52.130006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:52.129972 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25"] Nov 03 21:03:52.134306 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:03:52.134277 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bcb305_a8b0_421e_b79c_5f7929f3b5dc.slice/crio-8003c7bc9bfe91d026d370b9d267c435865f5bd1af78b8f40248a6334059273b WatchSource:0}: Error finding container 8003c7bc9bfe91d026d370b9d267c435865f5bd1af78b8f40248a6334059273b: Status 404 returned error can't find the container with id 8003c7bc9bfe91d026d370b9d267c435865f5bd1af78b8f40248a6334059273b Nov 03 21:03:52.245452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:52.245419 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" event={"ID":"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc","Type":"ContainerStarted","Data":"8003c7bc9bfe91d026d370b9d267c435865f5bd1af78b8f40248a6334059273b"} Nov 03 21:03:53.257166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:53.257126 5932 generic.go:356] "Generic (PLEG): container finished" podID="b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" containerID="2d8bddc041c9f5e84b9aeb50751b20684e46a032e396aa24fdf593326f910c8f" exitCode=0 Nov 03 21:03:53.257512 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:53.257206 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" event={"ID":"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc","Type":"ContainerDied","Data":"2d8bddc041c9f5e84b9aeb50751b20684e46a032e396aa24fdf593326f910c8f"} Nov 03 21:03:54.441719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.441694 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:54.504251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.504213 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca\") pod \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " Nov 03 21:03:54.504396 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.504353 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9\" (UniqueName: \"kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9\") pod \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " Nov 03 21:03:54.504505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.504483 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca" pod "b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" (UID: "b9bcb305-a8b0-421e-b79c-5f7929f3b5dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:03:54.504547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.504497 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca\") pod \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\" (UID: \"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc\") " Nov 03 21:03:54.505146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.505125 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:54.506490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.506460 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca" pod "b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" (UID: "b9bcb305-a8b0-421e-b79c-5f7929f3b5dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:03:54.506633 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.506604 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9" pod "b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" (UID: "b9bcb305-a8b0-421e-b79c-5f7929f3b5dc"). InnerVolumeSpecName "kube-api-access-d9rf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:03:54.606234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.606202 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:54.606234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:54.606232 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9rf9\" (UniqueName: \"kubernetes.io/projected/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc-kube-api-access-d9rf9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:03:55.275633 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:55.275589 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" Nov 03 21:03:55.275918 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:55.275618 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25" event={"ID":"b9bcb305-a8b0-421e-b79c-5f7929f3b5dc","Type":"ContainerDied","Data":"8003c7bc9bfe91d026d370b9d267c435865f5bd1af78b8f40248a6334059273b"} Nov 03 21:03:55.276060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:55.275945 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8003c7bc9bfe91d026d370b9d267c435865f5bd1af78b8f40248a6334059273b" Nov 03 21:03:56.504008 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:56.503975 5932 scope.go:122] "RemoveContainer" containerID="ace4cbb5431e5524119e071c6ee15633a77b65a6234aa245070476dd153cf2e9" Nov 03 21:03:56.515467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:56.515446 5932 scope.go:122] "RemoveContainer" containerID="881c6c350dcdd80c3ab860b3908c0781888838d02557879d7fe3bd4e38aed860" Nov 03 21:03:56.525728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:03:56.525705 5932 scope.go:122] "RemoveContainer" containerID="cddce299fa214f7f2725e901733c12eebcbab3f9fdca34f262b7b49e5ed79d37" Nov 03 21:04:14.153673 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:14.153618 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:04:21.116595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:21.116557 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz"] Nov 03 21:04:21.162076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:21.162039 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-5b12d6-c6xqz"] Nov 03 21:04:22.054901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:22.054868 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf8266f-0fdd-41d7-8e54-20eacf383629" path="/var/lib/kubelet/pods/2bf8266f-0fdd-41d7-8e54-20eacf383629/volumes" Nov 03 21:04:28.756820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:28.756785 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq"] Nov 03 21:04:28.757649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:28.757598 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" containerID="cri-o://0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37" gracePeriod=30 Nov 03 21:04:32.830251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:32.830160 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs"] Nov 03 21:04:32.949063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:32.949023 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26"] Nov 03 21:04:34.087872 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.087840 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 21:04:34.140878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.140840 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.140945 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.140975 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.140994 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141072 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141341 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141112 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141341 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141153 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:34.141341 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141273 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:34.141341 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141284 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141365 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141611 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx\") pod \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\" (UID: \"9b2828a1-58e3-4cfa-8ac7-760aab663beb\") " Nov 03 21:04:34.141898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.141873 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:34.142037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.142009 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:34.142649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.142610 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:34.142799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.142705 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.142942 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.142901 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.142942 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.142932 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.143121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.142951 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.144399 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.144376 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:04:34.144568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.144551 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:34.144682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.144607 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:04:34.145019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.144989 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx" pod "9b2828a1-58e3-4cfa-8ac7-760aab663beb" (UID: "9b2828a1-58e3-4cfa-8ac7-760aab663beb"). InnerVolumeSpecName "kube-api-access-5slhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:34.244473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.244426 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5slhx\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-kube-api-access-5slhx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.244473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.244465 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.244473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.244476 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.244473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.244487 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.244949 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.244496 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2828a1-58e3-4cfa-8ac7-760aab663beb-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:34.793483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.793444 5932 generic.go:356] "Generic (PLEG): container finished" podID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerID="0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37" exitCode=0 Nov 03 21:04:34.793754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.793511 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" Nov 03 21:04:34.793754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.793541 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" event={"ID":"9b2828a1-58e3-4cfa-8ac7-760aab663beb","Type":"ContainerDied","Data":"0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37"} Nov 03 21:04:34.793754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.793604 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq" event={"ID":"9b2828a1-58e3-4cfa-8ac7-760aab663beb","Type":"ContainerDied","Data":"26b6ba946248469c9ac7f206bcc828383ebafed1e140225557262f9fb956274f"} Nov 03 21:04:34.793754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.793647 5932 scope.go:122] "RemoveContainer" containerID="0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37" Nov 03 21:04:34.807589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.807561 5932 scope.go:122] "RemoveContainer" containerID="ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035" Nov 03 21:04:34.819638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.819598 5932 scope.go:122] "RemoveContainer" containerID="0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37" Nov 03 21:04:34.820017 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:34.819995 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37\": container with ID starting with 0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37 not found: ID does not exist" containerID="0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37" Nov 03 21:04:34.820117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.820033 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37"} err="failed to get container status \"0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37\": rpc error: code = NotFound desc = could not find container \"0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37\": container with ID starting with 0c685878d3b7b0e4501f6258e1e2d6f15afd26f626a113dd8b2895d2732aab37 not found: ID does not exist" Nov 03 21:04:34.820117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.820061 5932 scope.go:122] "RemoveContainer" containerID="ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035" Nov 03 21:04:34.820360 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:34.820337 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035\": container with ID starting with ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035 not found: ID does not exist" containerID="ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035" Nov 03 21:04:34.820416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.820367 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035"} err="failed to get container status \"ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035\": rpc error: code = NotFound desc = could not find container \"ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035\": container with ID starting with ef7d04afaa91ed0d273b4f68e93f6ebe552b938de4c89161d28665edb8e74035 not found: ID does not exist" Nov 03 21:04:34.958160 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:34.958124 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq"] Nov 03 21:04:35.000925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.000883 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-5b12d6-openshift-default-7dbfb684f9-x5psq"] Nov 03 21:04:35.806009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.805977 5932 generic.go:356] "Generic (PLEG): container finished" podID="c9e11e9a-8524-4256-b778-5504f5167180" containerID="0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323" exitCode=0 Nov 03 21:04:35.806589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.806032 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" event={"ID":"c9e11e9a-8524-4256-b778-5504f5167180","Type":"ContainerDied","Data":"0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323"} Nov 03 21:04:35.806589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.806499 5932 scope.go:122] "RemoveContainer" containerID="0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323" Nov 03 21:04:35.808362 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.808331 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:35.808451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.808406 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:35.808451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:35.808445 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:04:36.056010 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:36.055930 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" path="/var/lib/kubelet/pods/9b2828a1-58e3-4cfa-8ac7-760aab663beb/volumes" Nov 03 21:04:36.817612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:36.817572 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" event={"ID":"c9e11e9a-8524-4256-b778-5504f5167180","Type":"ContainerStarted","Data":"cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b"} Nov 03 21:04:37.816606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:37.816573 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:04:37.816806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:37.816751 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 21:04:37.816863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:37.816802 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 21:04:37.834651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:37.830994 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" podUID="2fba486a-efd8-47e1-893a-d408c3daeee9" containerName="payload-processing" containerID="cri-o://983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d" gracePeriod=25 Nov 03 21:04:37.950386 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:37.950343 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" podUID="2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" containerName="payload-pre-processing" containerID="cri-o://e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0" gracePeriod=25 Nov 03 21:04:38.212118 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.212085 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 21:04:38.292991 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.292957 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 21:04:38.357687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.356809 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume\") pod \"2fba486a-efd8-47e1-893a-d408c3daeee9\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " Nov 03 21:04:38.357687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.357120 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872\" (UniqueName: \"kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872\") pod \"2fba486a-efd8-47e1-893a-d408c3daeee9\" (UID: \"2fba486a-efd8-47e1-893a-d408c3daeee9\") " Nov 03 21:04:38.357687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.357406 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume" pod "2fba486a-efd8-47e1-893a-d408c3daeee9" (UID: "2fba486a-efd8-47e1-893a-d408c3daeee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:38.357940 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.357843 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fba486a-efd8-47e1-893a-d408c3daeee9-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:38.359484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.359450 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872" pod "2fba486a-efd8-47e1-893a-d408c3daeee9" (UID: "2fba486a-efd8-47e1-893a-d408c3daeee9"). InnerVolumeSpecName "kube-api-access-gf872". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:38.459720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.459643 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume\") pod \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " Nov 03 21:04:38.459991 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.459806 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54\" (UniqueName: \"kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54\") pod \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\" (UID: \"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5\") " Nov 03 21:04:38.460312 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.460287 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume" pod "2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" (UID: "2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:38.460547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.460525 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:38.460680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.460551 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf872\" (UniqueName: \"kubernetes.io/projected/2fba486a-efd8-47e1-893a-d408c3daeee9-kube-api-access-gf872\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:38.462942 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.462902 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54" pod "2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" (UID: "2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5"). InnerVolumeSpecName "kube-api-access-4lz54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:38.562565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.562518 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lz54\" (UniqueName: \"kubernetes.io/projected/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5-kube-api-access-4lz54\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:38.816512 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.816478 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 21:04:38.816742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.816530 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 21:04:38.856119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.856082 5932 generic.go:356] "Generic (PLEG): container finished" podID="2fba486a-efd8-47e1-893a-d408c3daeee9" containerID="983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d" exitCode=0 Nov 03 21:04:38.856589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.856138 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" Nov 03 21:04:38.856589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.856159 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" event={"ID":"2fba486a-efd8-47e1-893a-d408c3daeee9","Type":"ContainerDied","Data":"983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d"} Nov 03 21:04:38.856589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.856230 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs" event={"ID":"2fba486a-efd8-47e1-893a-d408c3daeee9","Type":"ContainerDied","Data":"c27945c627f2b0578b97ae1b758018f0d71a29ef132bce35f4d9f8b3ecfb5a85"} Nov 03 21:04:38.856589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.856261 5932 scope.go:122] "RemoveContainer" containerID="983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d" Nov 03 21:04:38.858284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.858199 5932 generic.go:356] "Generic (PLEG): container finished" podID="2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" containerID="e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0" exitCode=0 Nov 03 21:04:38.858284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.858259 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" Nov 03 21:04:38.858484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.858257 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" event={"ID":"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5","Type":"ContainerDied","Data":"e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0"} Nov 03 21:04:38.858484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.858399 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26" event={"ID":"2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5","Type":"ContainerDied","Data":"4ea62e754dc0c3d8355425aac6cc2b97bf3d933cd562a0141a1d287238700c43"} Nov 03 21:04:38.882612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.882583 5932 scope.go:122] "RemoveContainer" containerID="983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d" Nov 03 21:04:38.883015 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:38.882992 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d\": container with ID starting with 983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d not found: ID does not exist" containerID="983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d" Nov 03 21:04:38.883054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.883028 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d"} err="failed to get container status \"983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d\": rpc error: code = NotFound desc = could not find container \"983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d\": container with ID starting with 983de00e6c28ba5260b10b2c7f9e7ed37c06dc96b0533e3f493c42aa7dc08e9d not found: ID does not exist" Nov 03 21:04:38.883054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.883047 5932 scope.go:122] "RemoveContainer" containerID="e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0" Nov 03 21:04:38.894114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.894091 5932 scope.go:122] "RemoveContainer" containerID="e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0" Nov 03 21:04:38.894453 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:38.894430 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0\": container with ID starting with e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0 not found: ID does not exist" containerID="e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0" Nov 03 21:04:38.894530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:38.894461 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0"} err="failed to get container status \"e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0\": rpc error: code = NotFound desc = could not find container \"e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0\": container with ID starting with e1bbf48de9c240009815594632823474d0252ea3a1252843f96a428d8389fdb0 not found: ID does not exist" Nov 03 21:04:39.012898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:39.012868 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs"] Nov 03 21:04:39.016828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:39.016804 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e427e7-67b84f9f7-fn6bs"] Nov 03 21:04:39.119021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:39.118936 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26"] Nov 03 21:04:39.146130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:39.146085 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-e427e7-788f9dbc8c-wrk26"] Nov 03 21:04:39.865085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:39.853763 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 21:04:39.865085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:39.853840 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 21:04:40.061983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:40.061947 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" path="/var/lib/kubelet/pods/2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5/volumes" Nov 03 21:04:40.062445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:40.062430 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fba486a-efd8-47e1-893a-d408c3daeee9" path="/var/lib/kubelet/pods/2fba486a-efd8-47e1-893a-d408c3daeee9/volumes" Nov 03 21:04:40.816502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:40.816454 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 21:04:40.816722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:40.816536 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 21:04:41.816576 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:41.816539 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Nov 03 21:04:41.817013 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:41.816598 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Nov 03 21:04:42.822033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:42.821993 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:04:42.822558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:42.822524 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:04:42.823569 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:42.823547 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:04:44.153395 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:44.153351 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:04:44.505471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:44.505379 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p"] Nov 03 21:04:44.624912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:44.624862 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4"] Nov 03 21:04:44.983496 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:44.981118 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2"] Nov 03 21:04:44.984977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:44.984064 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" podUID="8020df72-b51d-4acb-9816-46ee6d01ce6c" containerName="maas-api" containerID="cri-o://389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01" gracePeriod=30 Nov 03 21:04:45.423153 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.419086 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:04:45.496500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.496464 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca\") pod \"8020df72-b51d-4acb-9816-46ee6d01ce6c\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " Nov 03 21:04:45.496707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.496606 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs\") pod \"8020df72-b51d-4acb-9816-46ee6d01ce6c\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " Nov 03 21:04:45.496792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.496717 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j\" (UniqueName: \"kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j\") pod \"8020df72-b51d-4acb-9816-46ee6d01ce6c\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " Nov 03 21:04:45.496792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.496765 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\") pod \"8020df72-b51d-4acb-9816-46ee6d01ce6c\" (UID: \"8020df72-b51d-4acb-9816-46ee6d01ce6c\") " Nov 03 21:04:45.496955 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.496929 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca" pod "8020df72-b51d-4acb-9816-46ee6d01ce6c" (UID: "8020df72-b51d-4acb-9816-46ee6d01ce6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:45.497394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.497374 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8020df72-b51d-4acb-9816-46ee6d01ce6c-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:45.498878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.498843 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls" pod "8020df72-b51d-4acb-9816-46ee6d01ce6c" (UID: "8020df72-b51d-4acb-9816-46ee6d01ce6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:45.498992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.498877 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs" pod "8020df72-b51d-4acb-9816-46ee6d01ce6c" (UID: "8020df72-b51d-4acb-9816-46ee6d01ce6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:04:45.498992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.498918 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j" pod "8020df72-b51d-4acb-9816-46ee6d01ce6c" (UID: "8020df72-b51d-4acb-9816-46ee6d01ce6c"). InnerVolumeSpecName "kube-api-access-dpt4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:45.598786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.598687 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:45.598786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.598729 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpt4j\" (UniqueName: \"kubernetes.io/projected/8020df72-b51d-4acb-9816-46ee6d01ce6c-kube-api-access-dpt4j\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:45.598786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:45.598747 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8020df72-b51d-4acb-9816-46ee6d01ce6c-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:46.259877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.258116 5932 generic.go:356] "Generic (PLEG): container finished" podID="8020df72-b51d-4acb-9816-46ee6d01ce6c" containerID="389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01" exitCode=0 Nov 03 21:04:46.260545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.260517 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" Nov 03 21:04:46.268868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.268794 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" event={"ID":"8020df72-b51d-4acb-9816-46ee6d01ce6c","Type":"ContainerDied","Data":"389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01"} Nov 03 21:04:46.275199 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.275161 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2" event={"ID":"8020df72-b51d-4acb-9816-46ee6d01ce6c","Type":"ContainerDied","Data":"8c550024e5ee3a55c71ea061b73bf22d82a8f2c9329df995376a354998c0ce07"} Nov 03 21:04:46.275472 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.275270 5932 scope.go:122] "RemoveContainer" containerID="389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01" Nov 03 21:04:46.292677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.291793 5932 scope.go:122] "RemoveContainer" containerID="389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01" Nov 03 21:04:46.292677 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:46.292140 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01\": container with ID starting with 389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01 not found: ID does not exist" containerID="389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01" Nov 03 21:04:46.292677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.292173 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01"} err="failed to get container status \"389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01\": rpc error: code = NotFound desc = could not find container \"389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01\": container with ID starting with 389bfc92d8e24404886cdcc8df8decf75aa52368c39f2117c80818f3d8ddce01 not found: ID does not exist" Nov 03 21:04:46.437842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.437799 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2"] Nov 03 21:04:46.442713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:46.442680 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-af2e8a-5f7559d848-xmkh2"] Nov 03 21:04:47.600181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:47.600103 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84"] Nov 03 21:04:47.604998 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:47.604968 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-af2e8a-pdz84"] Nov 03 21:04:48.054172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:48.054138 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8020df72-b51d-4acb-9816-46ee6d01ce6c" path="/var/lib/kubelet/pods/8020df72-b51d-4acb-9816-46ee6d01ce6c/volumes" Nov 03 21:04:48.054747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:48.054731 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e54f9c4a-b1f4-488c-a3a6-b767d5d8393c" path="/var/lib/kubelet/pods/e54f9c4a-b1f4-488c-a3a6-b767d5d8393c/volumes" Nov 03 21:04:49.505801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.505737 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" podUID="5af792c6-f691-489a-a846-060155aa0ce0" containerName="payload-processing" containerID="cri-o://15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9" gracePeriod=25 Nov 03 21:04:49.625954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.625907 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" podUID="bb171c86-c01b-4785-ad64-da20394c9879" containerName="payload-pre-processing" containerID="cri-o://ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa" gracePeriod=25 Nov 03 21:04:49.830949 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.830923 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:04:49.912023 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.911990 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx\" (UniqueName: \"kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx\") pod \"5af792c6-f691-489a-a846-060155aa0ce0\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " Nov 03 21:04:49.912234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.912088 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume\") pod \"5af792c6-f691-489a-a846-060155aa0ce0\" (UID: \"5af792c6-f691-489a-a846-060155aa0ce0\") " Nov 03 21:04:49.912511 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.912480 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume" pod "5af792c6-f691-489a-a846-060155aa0ce0" (UID: "5af792c6-f691-489a-a846-060155aa0ce0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:49.914773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.914739 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx" pod "5af792c6-f691-489a-a846-060155aa0ce0" (UID: "5af792c6-f691-489a-a846-060155aa0ce0"). InnerVolumeSpecName "kube-api-access-fvlvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:49.944562 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:49.944533 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:04:50.013328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.013291 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume\") pod \"bb171c86-c01b-4785-ad64-da20394c9879\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " Nov 03 21:04:50.013523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.013370 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x\" (UniqueName: \"kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x\") pod \"bb171c86-c01b-4785-ad64-da20394c9879\" (UID: \"bb171c86-c01b-4785-ad64-da20394c9879\") " Nov 03 21:04:50.013751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.013725 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume" pod "bb171c86-c01b-4785-ad64-da20394c9879" (UID: "bb171c86-c01b-4785-ad64-da20394c9879"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:04:50.013832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.013817 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvlvx\" (UniqueName: \"kubernetes.io/projected/5af792c6-f691-489a-a846-060155aa0ce0-kube-api-access-fvlvx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:50.013882 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.013835 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af792c6-f691-489a-a846-060155aa0ce0-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:50.015501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.015477 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x" pod "bb171c86-c01b-4785-ad64-da20394c9879" (UID: "bb171c86-c01b-4785-ad64-da20394c9879"). InnerVolumeSpecName "kube-api-access-84h6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:04:50.115061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.115024 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb171c86-c01b-4785-ad64-da20394c9879-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:50.115061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.115059 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84h6x\" (UniqueName: \"kubernetes.io/projected/bb171c86-c01b-4785-ad64-da20394c9879-kube-api-access-84h6x\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:04:50.296273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.296231 5932 generic.go:356] "Generic (PLEG): container finished" podID="5af792c6-f691-489a-a846-060155aa0ce0" containerID="15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9" exitCode=0 Nov 03 21:04:50.296452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.296284 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" Nov 03 21:04:50.296452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.296324 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" event={"ID":"5af792c6-f691-489a-a846-060155aa0ce0","Type":"ContainerDied","Data":"15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9"} Nov 03 21:04:50.296452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.296383 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p" event={"ID":"5af792c6-f691-489a-a846-060155aa0ce0","Type":"ContainerDied","Data":"6be6fe109d89e48bf13e1bb2255256043818ccb8f8e2a28a5e08571e844af701"} Nov 03 21:04:50.296452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.296410 5932 scope.go:122] "RemoveContainer" containerID="15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9" Nov 03 21:04:50.297993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.297967 5932 generic.go:356] "Generic (PLEG): container finished" podID="bb171c86-c01b-4785-ad64-da20394c9879" containerID="ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa" exitCode=0 Nov 03 21:04:50.298132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.298017 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" Nov 03 21:04:50.298132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.298054 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" event={"ID":"bb171c86-c01b-4785-ad64-da20394c9879","Type":"ContainerDied","Data":"ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa"} Nov 03 21:04:50.298132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.298106 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4" event={"ID":"bb171c86-c01b-4785-ad64-da20394c9879","Type":"ContainerDied","Data":"811602b29d2be5a03a905444bfe7ae5310ddeab4db0b917125bfdc705aa0d849"} Nov 03 21:04:50.307989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.307967 5932 scope.go:122] "RemoveContainer" containerID="15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9" Nov 03 21:04:50.308337 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:50.308315 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9\": container with ID starting with 15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9 not found: ID does not exist" containerID="15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9" Nov 03 21:04:50.308422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.308350 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9"} err="failed to get container status \"15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9\": rpc error: code = NotFound desc = could not find container \"15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9\": container with ID starting with 15c8930d18780380eb1c0da4aeaf4b98f7eceee990c410c45e162258b60c47a9 not found: ID does not exist" Nov 03 21:04:50.308422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.308375 5932 scope.go:122] "RemoveContainer" containerID="ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa" Nov 03 21:04:50.319655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.319614 5932 scope.go:122] "RemoveContainer" containerID="ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa" Nov 03 21:04:50.320001 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:04:50.319980 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa\": container with ID starting with ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa not found: ID does not exist" containerID="ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa" Nov 03 21:04:50.320054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.320009 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa"} err="failed to get container status \"ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa\": rpc error: code = NotFound desc = could not find container \"ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa\": container with ID starting with ed215731720690e129f4a2f8126cd3ae0c2607497f073f0729416e3d5c643dfa not found: ID does not exist" Nov 03 21:04:50.455447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.455404 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4"] Nov 03 21:04:50.461239 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.461198 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-37c5b0-579ff8898c-sgxz4"] Nov 03 21:04:50.563351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.563315 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p"] Nov 03 21:04:50.571085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:50.571004 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-37c5b0-86c548fc67-z2m4p"] Nov 03 21:04:52.055354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:52.055316 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af792c6-f691-489a-a846-060155aa0ce0" path="/var/lib/kubelet/pods/5af792c6-f691-489a-a846-060155aa0ce0/volumes" Nov 03 21:04:52.055956 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:52.055939 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb171c86-c01b-4785-ad64-da20394c9879" path="/var/lib/kubelet/pods/bb171c86-c01b-4785-ad64-da20394c9879/volumes" Nov 03 21:04:56.022097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:56.022065 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:04:56.032061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:04:56.032033 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:05:04.366568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:04.366530 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs"] Nov 03 21:05:04.367130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:04.366803 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" containerID="cri-o://b9247ca5c52722e2541626e366ad1620bb45b0e6780144831bf134eae7dcb03e" gracePeriod=30 Nov 03 21:05:06.008716 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.008677 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx"] Nov 03 21:05:06.009402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009387 5932 cpu_manager.go:414] "containerMap: removing container" podUID="8020df72-b51d-4acb-9816-46ee6d01ce6c" containerName="maas-api" Nov 03 21:05:06.009448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009403 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8020df72-b51d-4acb-9816-46ee6d01ce6c" containerName="maas-api" Nov 03 21:05:06.009448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009415 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2fba486a-efd8-47e1-893a-d408c3daeee9" containerName="payload-processing" Nov 03 21:05:06.009448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009421 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fba486a-efd8-47e1-893a-d408c3daeee9" containerName="payload-processing" Nov 03 21:05:06.009448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009441 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" Nov 03 21:05:06.009448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009447 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009456 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" containerName="payload-pre-processing" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009464 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" containerName="payload-pre-processing" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009480 5932 cpu_manager.go:414] "containerMap: removing container" podUID="bb171c86-c01b-4785-ad64-da20394c9879" containerName="payload-pre-processing" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009485 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb171c86-c01b-4785-ad64-da20394c9879" containerName="payload-pre-processing" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009496 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5af792c6-f691-489a-a846-060155aa0ce0" containerName="payload-processing" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009501 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af792c6-f691-489a-a846-060155aa0ce0" containerName="payload-processing" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009506 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" containerName="revoke-keys" Nov 03 21:05:06.009609 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009511 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" containerName="revoke-keys" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009617 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9f7bfc-acbc-4554-9e5b-e8a2aab509c5" containerName="payload-pre-processing" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009644 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb171c86-c01b-4785-ad64-da20394c9879" containerName="payload-pre-processing" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009651 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af792c6-f691-489a-a846-060155aa0ce0" containerName="payload-processing" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009656 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" containerName="revoke-keys" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009666 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009675 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="8020df72-b51d-4acb-9816-46ee6d01ce6c" containerName="maas-api" Nov 03 21:05:06.009915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.009682 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fba486a-efd8-47e1-893a-d408c3daeee9" containerName="payload-processing" Nov 03 21:05:06.015504 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.015483 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.067660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.067594 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:05:06.067660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.067616 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:05:06.096885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.096827 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.097315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.097290 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnc4x\" (UniqueName: \"kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.097744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.097717 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.140360 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.139147 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx"] Nov 03 21:05:06.199041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.198998 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.199259 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.199137 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnc4x\" (UniqueName: \"kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.199348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.199330 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.199711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.199693 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.201746 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.201726 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.256667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.256617 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnc4x\" (UniqueName: \"kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x\") pod \"maas-api-revoke-keys-e2e-inf-a-878695-pfrjx\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.332547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.332434 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:06.353686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.353599 5932 patch_prober.go:29] interesting pod/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:05:06.353817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.353687 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:05:06.712820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:06.712784 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx"] Nov 03 21:05:06.718455 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:05:06.718429 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34272f6f_e3ef_45ef_9ce0_aa5a0d4940a6.slice/crio-e1a8543c4aad835311baf55cd347b1513a7840d7b4937732fb75b226210305c7 WatchSource:0}: Error finding container e1a8543c4aad835311baf55cd347b1513a7840d7b4937732fb75b226210305c7: Status 404 returned error can't find the container with id e1a8543c4aad835311baf55cd347b1513a7840d7b4937732fb75b226210305c7 Nov 03 21:05:07.492265 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:07.492164 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" event={"ID":"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6","Type":"ContainerStarted","Data":"aa278c2ebf20abf6d328501bb10c744a1d7cf4793ec0caf578263ab65f7f5978"} Nov 03 21:05:07.492785 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:07.492286 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" event={"ID":"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6","Type":"ContainerStarted","Data":"e1a8543c4aad835311baf55cd347b1513a7840d7b4937732fb75b226210305c7"} Nov 03 21:05:07.612933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:07.612875 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" podStartSLOduration=2.612860743 podStartE2EDuration="2.612860743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:07.611155158 +0000 UTC m=+2115.676396633" watchObservedRunningTime="2026-11-03 21:05:07.612860743 +0000 UTC m=+2115.678102216" Nov 03 21:05:08.502368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:08.502331 5932 generic.go:356] "Generic (PLEG): container finished" podID="34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" containerID="aa278c2ebf20abf6d328501bb10c744a1d7cf4793ec0caf578263ab65f7f5978" exitCode=0 Nov 03 21:05:08.502829 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:08.502412 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" event={"ID":"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6","Type":"ContainerDied","Data":"aa278c2ebf20abf6d328501bb10c744a1d7cf4793ec0caf578263ab65f7f5978"} Nov 03 21:05:09.515290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.515248 5932 generic.go:356] "Generic (PLEG): container finished" podID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerID="b9247ca5c52722e2541626e366ad1620bb45b0e6780144831bf134eae7dcb03e" exitCode=0 Nov 03 21:05:09.515703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.515301 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" event={"ID":"a847dd75-4df4-45dd-8511-7bde98fd7c92","Type":"ContainerDied","Data":"b9247ca5c52722e2541626e366ad1620bb45b0e6780144831bf134eae7dcb03e"} Nov 03 21:05:09.515703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.515368 5932 scope.go:122] "RemoveContainer" containerID="6792df5727ef0b3ef240dfcc45b089ce09a4b2e18edd1c066a212bef477ae7c9" Nov 03 21:05:09.690890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.689854 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:05:09.757211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757174 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.757373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757235 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.757373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757328 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.757553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757448 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.757553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757501 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.757553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757512 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:09.757553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757525 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:09.757553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757540 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.758005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757582 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.758005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757662 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.758005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757764 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:09.758005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757796 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert\") pod \"a847dd75-4df4-45dd-8511-7bde98fd7c92\" (UID: \"a847dd75-4df4-45dd-8511-7bde98fd7c92\") " Nov 03 21:05:09.758005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.757818 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:09.758275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.758185 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:09.758460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.758437 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a847dd75-4df4-45dd-8511-7bde98fd7c92-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.758566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.758464 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.758566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.758480 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.758566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.758493 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.758566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.758509 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.759940 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.759913 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:09.760054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.759980 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:09.760322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.760295 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:09.760453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.760437 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96" pod "a847dd75-4df4-45dd-8511-7bde98fd7c92" (UID: "a847dd75-4df4-45dd-8511-7bde98fd7c92"). InnerVolumeSpecName "kube-api-access-kll96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:09.770836 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.770814 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:09.859287 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.859251 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca\") pod \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " Nov 03 21:05:09.859502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.859384 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x\" (UniqueName: \"kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x\") pod \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " Nov 03 21:05:09.859576 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.859518 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca\") pod \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\" (UID: \"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6\") " Nov 03 21:05:09.859733 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.859705 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca" pod "34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" (UID: "34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:09.860111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.860084 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kll96\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-kube-api-access-kll96\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.860111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.860115 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.860303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.860129 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.860303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.860142 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.860303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.860154 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a847dd75-4df4-45dd-8511-7bde98fd7c92-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.863879 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.863853 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x" pod "34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" (UID: "34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6"). InnerVolumeSpecName "kube-api-access-qnc4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:09.863978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.863955 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca" pod "34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" (UID: "34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:09.962087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.962000 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnc4x\" (UniqueName: \"kubernetes.io/projected/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-kube-api-access-qnc4x\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:09.962087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:09.962034 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:10.524462 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.524429 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" Nov 03 21:05:10.524974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.524468 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs" event={"ID":"a847dd75-4df4-45dd-8511-7bde98fd7c92","Type":"ContainerDied","Data":"8e59fc2e5231e147952e961560122fd810836455658b7b8244cb41a70ebe10a6"} Nov 03 21:05:10.524974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.524528 5932 scope.go:122] "RemoveContainer" containerID="b9247ca5c52722e2541626e366ad1620bb45b0e6780144831bf134eae7dcb03e" Nov 03 21:05:10.526351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.526069 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" event={"ID":"34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6","Type":"ContainerDied","Data":"e1a8543c4aad835311baf55cd347b1513a7840d7b4937732fb75b226210305c7"} Nov 03 21:05:10.526351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.526091 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx" Nov 03 21:05:10.526351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.526115 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1a8543c4aad835311baf55cd347b1513a7840d7b4937732fb75b226210305c7" Nov 03 21:05:10.730094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.730004 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs"] Nov 03 21:05:10.736185 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:10.736150 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-af2e8a-openshift-default-d595b47f9-hlbfs"] Nov 03 21:05:12.055523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:12.055491 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" path="/var/lib/kubelet/pods/a847dd75-4df4-45dd-8511-7bde98fd7c92/volumes" Nov 03 21:05:14.153027 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:14.152990 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:05:16.105320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:16.105278 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9"] Nov 03 21:05:17.654646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:17.647351 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46"] Nov 03 21:05:17.700642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:17.698856 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-xmt46"] Nov 03 21:05:18.060511 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:18.060415 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44d28a6-142a-49e7-98d6-c1e3b8fca87c" path="/var/lib/kubelet/pods/a44d28a6-142a-49e7-98d6-c1e3b8fca87c/volumes" Nov 03 21:05:21.106281 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.106235 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" podUID="0985b16e-aac5-4ceb-80a3-ea2a54800271" containerName="payload-processing" containerID="cri-o://71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef" gracePeriod=25 Nov 03 21:05:21.397185 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.397158 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:05:21.466012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.465976 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z\") pod \"0985b16e-aac5-4ceb-80a3-ea2a54800271\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " Nov 03 21:05:21.466180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.466161 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume\") pod \"0985b16e-aac5-4ceb-80a3-ea2a54800271\" (UID: \"0985b16e-aac5-4ceb-80a3-ea2a54800271\") " Nov 03 21:05:21.466512 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.466487 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume" pod "0985b16e-aac5-4ceb-80a3-ea2a54800271" (UID: "0985b16e-aac5-4ceb-80a3-ea2a54800271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:21.466653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.466615 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0985b16e-aac5-4ceb-80a3-ea2a54800271-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.468061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.468038 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z" pod "0985b16e-aac5-4ceb-80a3-ea2a54800271" (UID: "0985b16e-aac5-4ceb-80a3-ea2a54800271"). InnerVolumeSpecName "kube-api-access-vdl2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:21.567819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.567778 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdl2z\" (UniqueName: \"kubernetes.io/projected/0985b16e-aac5-4ceb-80a3-ea2a54800271-kube-api-access-vdl2z\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:21.635587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.635550 5932 generic.go:356] "Generic (PLEG): container finished" podID="0985b16e-aac5-4ceb-80a3-ea2a54800271" containerID="71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef" exitCode=0 Nov 03 21:05:21.635828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.635608 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" Nov 03 21:05:21.635828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.635658 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" event={"ID":"0985b16e-aac5-4ceb-80a3-ea2a54800271","Type":"ContainerDied","Data":"71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef"} Nov 03 21:05:21.635828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.635717 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9" event={"ID":"0985b16e-aac5-4ceb-80a3-ea2a54800271","Type":"ContainerDied","Data":"90b3b7cd2264c71227279b811652c4f9871b177c6645ef9fbc50fcbd8aacd027"} Nov 03 21:05:21.635828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.635742 5932 scope.go:122] "RemoveContainer" containerID="71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef" Nov 03 21:05:21.647341 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.647323 5932 scope.go:122] "RemoveContainer" containerID="71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef" Nov 03 21:05:21.647643 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:21.647610 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef\": container with ID starting with 71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef not found: ID does not exist" containerID="71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef" Nov 03 21:05:21.647719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.647655 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef"} err="failed to get container status \"71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef\": rpc error: code = NotFound desc = could not find container \"71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef\": container with ID starting with 71503dca2f2eeea6ff99f77f0ee7161c9c3ffd769348682d97d7440900d7e8ef not found: ID does not exist" Nov 03 21:05:21.791068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.791028 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9"] Nov 03 21:05:21.795158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:21.795128 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-878695-5c7b4cdb9f-2b6n9"] Nov 03 21:05:22.053641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:22.053549 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0985b16e-aac5-4ceb-80a3-ea2a54800271" path="/var/lib/kubelet/pods/0985b16e-aac5-4ceb-80a3-ea2a54800271/volumes" Nov 03 21:05:37.424385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.424352 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9"] Nov 03 21:05:37.424895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.424552 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" podUID="fa645b1b-2011-4d56-9542-407a948d3c55" containerName="maas-api" containerID="cri-o://5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f" gracePeriod=30 Nov 03 21:05:37.768880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.768845 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 21:05:37.813111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.813075 5932 generic.go:356] "Generic (PLEG): container finished" podID="fa645b1b-2011-4d56-9542-407a948d3c55" containerID="5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f" exitCode=0 Nov 03 21:05:37.813316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.813121 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" Nov 03 21:05:37.813316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.813165 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" event={"ID":"fa645b1b-2011-4d56-9542-407a948d3c55","Type":"ContainerDied","Data":"5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f"} Nov 03 21:05:37.813316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.813228 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9" event={"ID":"fa645b1b-2011-4d56-9542-407a948d3c55","Type":"ContainerDied","Data":"64c3013fd387f7fb5d06ab6854394fa4737cb734d8c464fc3b3119c2c191fd4f"} Nov 03 21:05:37.813316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.813258 5932 scope.go:122] "RemoveContainer" containerID="5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f" Nov 03 21:05:37.824312 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.824289 5932 scope.go:122] "RemoveContainer" containerID="5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f" Nov 03 21:05:37.824655 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:37.824613 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f\": container with ID starting with 5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f not found: ID does not exist" containerID="5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f" Nov 03 21:05:37.824744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.824665 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f"} err="failed to get container status \"5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f\": rpc error: code = NotFound desc = could not find container \"5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f\": container with ID starting with 5344c6185fe25ce29d4365fc4c2782e88e83bd41753ea0761dd3582b9a30c68f not found: ID does not exist" Nov 03 21:05:37.880098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.880063 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk\" (UniqueName: \"kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk\") pod \"fa645b1b-2011-4d56-9542-407a948d3c55\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " Nov 03 21:05:37.880308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.880167 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca\") pod \"fa645b1b-2011-4d56-9542-407a948d3c55\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " Nov 03 21:05:37.880308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.880274 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs\") pod \"fa645b1b-2011-4d56-9542-407a948d3c55\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " Nov 03 21:05:37.880589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.880550 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\") pod \"fa645b1b-2011-4d56-9542-407a948d3c55\" (UID: \"fa645b1b-2011-4d56-9542-407a948d3c55\") " Nov 03 21:05:37.880914 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.880880 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca" pod "fa645b1b-2011-4d56-9542-407a948d3c55" (UID: "fa645b1b-2011-4d56-9542-407a948d3c55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:37.881710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.881682 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa645b1b-2011-4d56-9542-407a948d3c55-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:37.883462 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.883436 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs" pod "fa645b1b-2011-4d56-9542-407a948d3c55" (UID: "fa645b1b-2011-4d56-9542-407a948d3c55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:37.883568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.883468 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk" pod "fa645b1b-2011-4d56-9542-407a948d3c55" (UID: "fa645b1b-2011-4d56-9542-407a948d3c55"). InnerVolumeSpecName "kube-api-access-2wrkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:37.883568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.883484 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls" pod "fa645b1b-2011-4d56-9542-407a948d3c55" (UID: "fa645b1b-2011-4d56-9542-407a948d3c55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:37.990766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.985126 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wrkk\" (UniqueName: \"kubernetes.io/projected/fa645b1b-2011-4d56-9542-407a948d3c55-kube-api-access-2wrkk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:37.990766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.985156 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:37.990766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:37.985166 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa645b1b-2011-4d56-9542-407a948d3c55-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:38.246713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:38.240317 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9"] Nov 03 21:05:38.250408 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:38.250328 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2e3ea2-5fd8f95cd5-m5mt9"] Nov 03 21:05:40.053195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:40.053160 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa645b1b-2011-4d56-9542-407a948d3c55" path="/var/lib/kubelet/pods/fa645b1b-2011-4d56-9542-407a948d3c55/volumes" Nov 03 21:05:44.151250 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:44.151214 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:05:46.462199 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:46.462166 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq"] Nov 03 21:05:46.462698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:46.462370 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" containerID="cri-o://4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e" gracePeriod=30 Nov 03 21:05:47.377482 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.377449 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg"] Nov 03 21:05:47.378529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378511 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" Nov 03 21:05:47.378591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378533 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" Nov 03 21:05:47.378591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378560 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0985b16e-aac5-4ceb-80a3-ea2a54800271" containerName="payload-processing" Nov 03 21:05:47.378591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378568 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0985b16e-aac5-4ceb-80a3-ea2a54800271" containerName="payload-processing" Nov 03 21:05:47.378591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378586 5932 cpu_manager.go:414] "containerMap: removing container" podUID="34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" containerName="revoke-keys" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378593 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" containerName="revoke-keys" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378637 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378647 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378680 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378688 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378733 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fa645b1b-2011-4d56-9542-407a948d3c55" containerName="maas-api" Nov 03 21:05:47.378758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378740 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa645b1b-2011-4d56-9542-407a948d3c55" containerName="maas-api" Nov 03 21:05:47.378981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378893 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" Nov 03 21:05:47.378981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378916 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa645b1b-2011-4d56-9542-407a948d3c55" containerName="maas-api" Nov 03 21:05:47.378981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378926 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2828a1-58e3-4cfa-8ac7-760aab663beb" containerName="istio-proxy" Nov 03 21:05:47.378981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378938 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a847dd75-4df4-45dd-8511-7bde98fd7c92" containerName="istio-proxy" Nov 03 21:05:47.378981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378948 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0985b16e-aac5-4ceb-80a3-ea2a54800271" containerName="payload-processing" Nov 03 21:05:47.378981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.378956 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" containerName="revoke-keys" Nov 03 21:05:47.383076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.383056 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.434187 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.434154 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:05:47.434602 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.434581 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:05:47.538645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.535750 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg"] Nov 03 21:05:47.538645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.536852 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.538645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.537081 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdchn\" (UniqueName: \"kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.538645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.537137 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.644089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.643987 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdchn\" (UniqueName: \"kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.644089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.644042 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.644351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.644143 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.644803 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.644776 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.646515 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.646489 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.703749 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.703725 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdchn\" (UniqueName: \"kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:47.998466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:47.998391 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:48.272555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.272455 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d"] Nov 03 21:05:48.280933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.280901 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.327858 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.327824 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d"] Nov 03 21:05:48.354206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.354167 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.354390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.354359 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8sbx\" (UniqueName: \"kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.354453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.354422 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.392323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.392296 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg"] Nov 03 21:05:48.455933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.455898 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8sbx\" (UniqueName: \"kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.456078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.456006 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.456241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.456225 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.456945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.456920 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.458735 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.458715 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.517045 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.517017 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8sbx\" (UniqueName: \"kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.596031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.595943 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:48.910276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.910241 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj"] Nov 03 21:05:48.910466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.910440 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" containerID="cri-o://86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e" gracePeriod=30 Nov 03 21:05:48.921151 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.921113 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" event={"ID":"ce949650-dc23-4e4d-9865-cd44ab218709","Type":"ContainerStarted","Data":"201c92ef65887be06629ca9f76b12ed17448ae3049699f6d14194e5a5af9950a"} Nov 03 21:05:48.921308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.921184 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" event={"ID":"ce949650-dc23-4e4d-9865-cd44ab218709","Type":"ContainerStarted","Data":"0f2f1574d8dd3e1a35191ab9661c9d0c2f225a6a090b693774d2163d5f067f47"} Nov 03 21:05:48.980675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:48.980642 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d"] Nov 03 21:05:48.985710 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:05:48.985681 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf87bd98_07c6_49fc_8880_faeb56ed372c.slice/crio-d67bff8e2faf595f1efba55d34775afc8a870d4fd87ace6e061bf334b8678643 WatchSource:0}: Error finding container d67bff8e2faf595f1efba55d34775afc8a870d4fd87ace6e061bf334b8678643: Status 404 returned error can't find the container with id d67bff8e2faf595f1efba55d34775afc8a870d4fd87ace6e061bf334b8678643 Nov 03 21:05:49.183550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:49.183450 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" podStartSLOduration=2.183435555 podStartE2EDuration="2.183435555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:49.181876332 +0000 UTC m=+2157.247117806" watchObservedRunningTime="2026-11-03 21:05:49.183435555 +0000 UTC m=+2157.248677027" Nov 03 21:05:49.933109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:49.933066 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" event={"ID":"bf87bd98-07c6-49fc-8880-faeb56ed372c","Type":"ContainerStarted","Data":"cac75c4a20ab797f29fe6f3c89a7340d516e8f30b8074d294bd8e17f71bac40a"} Nov 03 21:05:49.933606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:49.933141 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" event={"ID":"bf87bd98-07c6-49fc-8880-faeb56ed372c","Type":"ContainerStarted","Data":"d67bff8e2faf595f1efba55d34775afc8a870d4fd87ace6e061bf334b8678643"} Nov 03 21:05:49.935057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:49.935023 5932 generic.go:356] "Generic (PLEG): container finished" podID="ce949650-dc23-4e4d-9865-cd44ab218709" containerID="201c92ef65887be06629ca9f76b12ed17448ae3049699f6d14194e5a5af9950a" exitCode=0 Nov 03 21:05:49.935189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:49.935073 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" event={"ID":"ce949650-dc23-4e4d-9865-cd44ab218709","Type":"ContainerDied","Data":"201c92ef65887be06629ca9f76b12ed17448ae3049699f6d14194e5a5af9950a"} Nov 03 21:05:50.059579 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:50.059510 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" podStartSLOduration=2.059485946 podStartE2EDuration="2.059485946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:05:50.043095381 +0000 UTC m=+2158.108337119" watchObservedRunningTime="2026-11-03 21:05:50.059485946 +0000 UTC m=+2158.124727420" Nov 03 21:05:50.944782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:50.944750 5932 generic.go:356] "Generic (PLEG): container finished" podID="bf87bd98-07c6-49fc-8880-faeb56ed372c" containerID="cac75c4a20ab797f29fe6f3c89a7340d516e8f30b8074d294bd8e17f71bac40a" exitCode=0 Nov 03 21:05:50.945266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:50.944839 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" event={"ID":"bf87bd98-07c6-49fc-8880-faeb56ed372c","Type":"ContainerDied","Data":"cac75c4a20ab797f29fe6f3c89a7340d516e8f30b8074d294bd8e17f71bac40a"} Nov 03 21:05:51.091575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.091544 5932 patch_prober.go:29] interesting pod/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:05:51.091747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.091599 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:05:51.110099 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.110077 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:51.191990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.191958 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca\") pod \"ce949650-dc23-4e4d-9865-cd44ab218709\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " Nov 03 21:05:51.192187 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.192013 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn\" (UniqueName: \"kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn\") pod \"ce949650-dc23-4e4d-9865-cd44ab218709\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " Nov 03 21:05:51.192187 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.192176 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca\") pod \"ce949650-dc23-4e4d-9865-cd44ab218709\" (UID: \"ce949650-dc23-4e4d-9865-cd44ab218709\") " Nov 03 21:05:51.192354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.192324 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca" pod "ce949650-dc23-4e4d-9865-cd44ab218709" (UID: "ce949650-dc23-4e4d-9865-cd44ab218709"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:51.192721 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.192699 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.194138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.194103 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn" pod "ce949650-dc23-4e4d-9865-cd44ab218709" (UID: "ce949650-dc23-4e4d-9865-cd44ab218709"). InnerVolumeSpecName "kube-api-access-qdchn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:51.194242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.194198 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca" pod "ce949650-dc23-4e4d-9865-cd44ab218709" (UID: "ce949650-dc23-4e4d-9865-cd44ab218709"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:51.294502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.294468 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdchn\" (UniqueName: \"kubernetes.io/projected/ce949650-dc23-4e4d-9865-cd44ab218709-kube-api-access-qdchn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.294502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.294500 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce949650-dc23-4e4d-9865-cd44ab218709-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.748115 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.748088 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 21:05:51.902142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902107 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902150 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902270 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902340 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902388 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902428 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902468 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902487 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:51.902618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902534 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:51.902618 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902576 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902638 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket\") pod \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\" (UID: \"5fcdb585-bf18-4dd5-a2dd-45a14c591520\") " Nov 03 21:05:51.902934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902712 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:51.902934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.902895 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:51.903340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.903315 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:51.903430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.903346 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.903430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.903365 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.903430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.903379 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.903430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.903391 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:51.905566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.905525 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:51.905748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.905580 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "kube-api-access-7xhbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:51.905947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.905922 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:51.906738 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.906710 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo" pod "5fcdb585-bf18-4dd5-a2dd-45a14c591520" (UID: "5fcdb585-bf18-4dd5-a2dd-45a14c591520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:51.966203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.966167 5932 generic.go:356] "Generic (PLEG): container finished" podID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerID="4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e" exitCode=0 Nov 03 21:05:51.966583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.966252 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" event={"ID":"5fcdb585-bf18-4dd5-a2dd-45a14c591520","Type":"ContainerDied","Data":"4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e"} Nov 03 21:05:51.966583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.966320 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" event={"ID":"5fcdb585-bf18-4dd5-a2dd-45a14c591520","Type":"ContainerDied","Data":"aa71f062e615f1083ceef919a5a4092e18dfaf687b37b3180619ee4d5440ede6"} Nov 03 21:05:51.966583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.966342 5932 scope.go:122] "RemoveContainer" containerID="4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e" Nov 03 21:05:51.966583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.966477 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq" Nov 03 21:05:51.968519 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.968494 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" Nov 03 21:05:51.968723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.968506 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg" event={"ID":"ce949650-dc23-4e4d-9865-cd44ab218709","Type":"ContainerDied","Data":"0f2f1574d8dd3e1a35191ab9661c9d0c2f225a6a090b693774d2163d5f067f47"} Nov 03 21:05:51.968847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.968833 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f2f1574d8dd3e1a35191ab9661c9d0c2f225a6a090b693774d2163d5f067f47" Nov 03 21:05:51.981551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.981522 5932 scope.go:122] "RemoveContainer" containerID="57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09" Nov 03 21:05:51.998136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.998105 5932 scope.go:122] "RemoveContainer" containerID="4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e" Nov 03 21:05:51.998653 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:51.998582 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e\": container with ID starting with 4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e not found: ID does not exist" containerID="4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e" Nov 03 21:05:51.998743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.998685 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e"} err="failed to get container status \"4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e\": rpc error: code = NotFound desc = could not find container \"4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e\": container with ID starting with 4975926d3c5ac7e7c7d14d72c4cf7b89912480f58ed8b110243346fa6382a26e not found: ID does not exist" Nov 03 21:05:51.998743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.998711 5932 scope.go:122] "RemoveContainer" containerID="57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09" Nov 03 21:05:51.999047 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:51.999015 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09\": container with ID starting with 57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09 not found: ID does not exist" containerID="57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09" Nov 03 21:05:51.999122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:51.999058 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09"} err="failed to get container status \"57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09\": rpc error: code = NotFound desc = could not find container \"57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09\": container with ID starting with 57632d65b390f5d6bd3840f04727da26a447e8003af8613b4f25245638404e09 not found: ID does not exist" Nov 03 21:05:52.005815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.005254 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xhbn\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-kube-api-access-7xhbn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.005815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.005291 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.005815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.005307 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.005815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.005323 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fcdb585-bf18-4dd5-a2dd-45a14c591520-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.005815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.005337 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fcdb585-bf18-4dd5-a2dd-45a14c591520-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.148710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.148684 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:52.171412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.171376 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq"] Nov 03 21:05:52.174902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.174878 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-e427e7-openshift-default-64695b9575-prvdq"] Nov 03 21:05:52.309302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.309207 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca\") pod \"bf87bd98-07c6-49fc-8880-faeb56ed372c\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " Nov 03 21:05:52.309491 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.309447 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx\" (UniqueName: \"kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx\") pod \"bf87bd98-07c6-49fc-8880-faeb56ed372c\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " Nov 03 21:05:52.309590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.309556 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca" pod "bf87bd98-07c6-49fc-8880-faeb56ed372c" (UID: "bf87bd98-07c6-49fc-8880-faeb56ed372c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:52.309705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.309576 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca\") pod \"bf87bd98-07c6-49fc-8880-faeb56ed372c\" (UID: \"bf87bd98-07c6-49fc-8880-faeb56ed372c\") " Nov 03 21:05:52.310333 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.310303 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.311883 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.311853 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca" pod "bf87bd98-07c6-49fc-8880-faeb56ed372c" (UID: "bf87bd98-07c6-49fc-8880-faeb56ed372c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:05:52.312201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.312168 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx" pod "bf87bd98-07c6-49fc-8880-faeb56ed372c" (UID: "bf87bd98-07c6-49fc-8880-faeb56ed372c"). InnerVolumeSpecName "kube-api-access-l8sbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:52.411360 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.411329 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf87bd98-07c6-49fc-8880-faeb56ed372c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.411360 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.411358 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8sbx\" (UniqueName: \"kubernetes.io/projected/bf87bd98-07c6-49fc-8880-faeb56ed372c-kube-api-access-l8sbx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:52.978214 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.978183 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" Nov 03 21:05:52.978670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.978181 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d" event={"ID":"bf87bd98-07c6-49fc-8880-faeb56ed372c","Type":"ContainerDied","Data":"d67bff8e2faf595f1efba55d34775afc8a870d4fd87ace6e061bf334b8678643"} Nov 03 21:05:52.978670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:52.978340 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d67bff8e2faf595f1efba55d34775afc8a870d4fd87ace6e061bf334b8678643" Nov 03 21:05:54.056469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.056434 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" path="/var/lib/kubelet/pods/5fcdb585-bf18-4dd5-a2dd-45a14c591520/volumes" Nov 03 21:05:54.192165 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.192096 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 21:05:54.333351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333311 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333367 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333440 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333479 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333663 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333675 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:54.333773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333695 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:54.333773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333729 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333836 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333916 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.333995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.333945 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data\") pod \"82786355-cc14-4d9f-9912-e1c13d7e3b62\" (UID: \"82786355-cc14-4d9f-9912-e1c13d7e3b62\") " Nov 03 21:05:54.334224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334067 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:54.334936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334427 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:05:54.334936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334697 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:54.334936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334743 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82786355-cc14-4d9f-9912-e1c13d7e3b62-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.334936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334792 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.334936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334810 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.334936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.334826 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.337874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.336529 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "kube-api-access-5kcpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:54.337874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.336901 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:05:54.337874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.337078 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:05:54.338403 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.338376 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo" pod "82786355-cc14-4d9f-9912-e1c13d7e3b62" (UID: "82786355-cc14-4d9f-9912-e1c13d7e3b62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:05:54.436303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.436267 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.436303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.436296 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.436513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.436312 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.436513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.436331 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kcpd\" (UniqueName: \"kubernetes.io/projected/82786355-cc14-4d9f-9912-e1c13d7e3b62-kube-api-access-5kcpd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.436513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.436342 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82786355-cc14-4d9f-9912-e1c13d7e3b62-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:05:54.998196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.998160 5932 generic.go:356] "Generic (PLEG): container finished" podID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerID="86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e" exitCode=0 Nov 03 21:05:54.998348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.998229 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" Nov 03 21:05:54.998348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.998242 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" event={"ID":"82786355-cc14-4d9f-9912-e1c13d7e3b62","Type":"ContainerDied","Data":"86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e"} Nov 03 21:05:54.998348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.998331 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj" event={"ID":"82786355-cc14-4d9f-9912-e1c13d7e3b62","Type":"ContainerDied","Data":"e981826c56f48c9359db48877b94b8a2406e2660acab5c57400e2da52269d8a1"} Nov 03 21:05:54.998486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:54.998355 5932 scope.go:122] "RemoveContainer" containerID="86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e" Nov 03 21:05:55.010170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:55.010148 5932 scope.go:122] "RemoveContainer" containerID="86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e" Nov 03 21:05:55.010475 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:05:55.010448 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e\": container with ID starting with 86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e not found: ID does not exist" containerID="86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e" Nov 03 21:05:55.010571 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:55.010486 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e"} err="failed to get container status \"86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e\": rpc error: code = NotFound desc = could not find container \"86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e\": container with ID starting with 86145367ab5ee52c5e33e4245d3b8862665756dcd568c9bab0efdca3f70f186e not found: ID does not exist" Nov 03 21:05:55.154886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:55.154851 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj"] Nov 03 21:05:55.201116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:55.201081 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-bjslj"] Nov 03 21:05:56.053367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:56.053334 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" path="/var/lib/kubelet/pods/82786355-cc14-4d9f-9912-e1c13d7e3b62/volumes" Nov 03 21:05:58.902616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:05:58.902574 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98"] Nov 03 21:06:01.486495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:01.486452 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm"] Nov 03 21:06:03.903100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:03.902997 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" podUID="152dd512-4484-489e-b01b-53ca4d512db6" containerName="payload-processing" containerID="cri-o://e0f3df577769177c362d4880d8da4cd5bf23a42aa127e8c37948eb078f2f9d97" gracePeriod=25 Nov 03 21:06:04.127163 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.127123 5932 generic.go:356] "Generic (PLEG): container finished" podID="152dd512-4484-489e-b01b-53ca4d512db6" containerID="e0f3df577769177c362d4880d8da4cd5bf23a42aa127e8c37948eb078f2f9d97" exitCode=0 Nov 03 21:06:04.127365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.127199 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" event={"ID":"152dd512-4484-489e-b01b-53ca4d512db6","Type":"ContainerDied","Data":"e0f3df577769177c362d4880d8da4cd5bf23a42aa127e8c37948eb078f2f9d97"} Nov 03 21:06:04.183741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.183715 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:06:04.210277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.210245 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume\") pod \"152dd512-4484-489e-b01b-53ca4d512db6\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " Nov 03 21:06:04.210469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.210458 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk\" (UniqueName: \"kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk\") pod \"152dd512-4484-489e-b01b-53ca4d512db6\" (UID: \"152dd512-4484-489e-b01b-53ca4d512db6\") " Nov 03 21:06:04.210583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.210563 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume" pod "152dd512-4484-489e-b01b-53ca4d512db6" (UID: "152dd512-4484-489e-b01b-53ca4d512db6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:04.210851 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.210833 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/152dd512-4484-489e-b01b-53ca4d512db6-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:04.212486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.212464 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk" pod "152dd512-4484-489e-b01b-53ca4d512db6" (UID: "152dd512-4484-489e-b01b-53ca4d512db6"). InnerVolumeSpecName "kube-api-access-zxfsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:04.311918 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:04.311882 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfsk\" (UniqueName: \"kubernetes.io/projected/152dd512-4484-489e-b01b-53ca4d512db6-kube-api-access-zxfsk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:05.137151 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:05.137113 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" event={"ID":"152dd512-4484-489e-b01b-53ca4d512db6","Type":"ContainerDied","Data":"d7a6144bd5764b6b96db7baa37d873565f46a39f0db83d3ec6b5c62dd240e5e2"} Nov 03 21:06:05.137594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:05.137175 5932 scope.go:122] "RemoveContainer" containerID="e0f3df577769177c362d4880d8da4cd5bf23a42aa127e8c37948eb078f2f9d97" Nov 03 21:06:05.137594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:05.137131 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98" Nov 03 21:06:05.350647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:05.337010 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98"] Nov 03 21:06:05.354749 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:05.354700 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-8448c9-775569857-7mz98"] Nov 03 21:06:06.060256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.060210 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="152dd512-4484-489e-b01b-53ca4d512db6" path="/var/lib/kubelet/pods/152dd512-4484-489e-b01b-53ca4d512db6/volumes" Nov 03 21:06:06.487773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.487731 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" podUID="48102ead-fefd-47a6-9f72-4ecee9cc7bda" containerName="payload-processing" containerID="cri-o://e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437" gracePeriod=25 Nov 03 21:06:06.775727 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.775703 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:06:06.875024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.874990 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume\") pod \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " Nov 03 21:06:06.875191 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.875108 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc\" (UniqueName: \"kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc\") pod \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\" (UID: \"48102ead-fefd-47a6-9f72-4ecee9cc7bda\") " Nov 03 21:06:06.875442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.875418 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume" pod "48102ead-fefd-47a6-9f72-4ecee9cc7bda" (UID: "48102ead-fefd-47a6-9f72-4ecee9cc7bda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:06.875726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.875709 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48102ead-fefd-47a6-9f72-4ecee9cc7bda-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:06.877203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.877178 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc" pod "48102ead-fefd-47a6-9f72-4ecee9cc7bda" (UID: "48102ead-fefd-47a6-9f72-4ecee9cc7bda"). InnerVolumeSpecName "kube-api-access-6r5kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:06.977419 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:06.977386 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r5kc\" (UniqueName: \"kubernetes.io/projected/48102ead-fefd-47a6-9f72-4ecee9cc7bda-kube-api-access-6r5kc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:07.186864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.186827 5932 generic.go:356] "Generic (PLEG): container finished" podID="48102ead-fefd-47a6-9f72-4ecee9cc7bda" containerID="e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437" exitCode=0 Nov 03 21:06:07.187073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.186884 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" Nov 03 21:06:07.187073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.186918 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" event={"ID":"48102ead-fefd-47a6-9f72-4ecee9cc7bda","Type":"ContainerDied","Data":"e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437"} Nov 03 21:06:07.187073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.186973 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm" event={"ID":"48102ead-fefd-47a6-9f72-4ecee9cc7bda","Type":"ContainerDied","Data":"b1593d94a11710e0e7ab098e5f5245b46f07ade17c1a595dd11f2a46cef37a62"} Nov 03 21:06:07.187073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.186998 5932 scope.go:122] "RemoveContainer" containerID="e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437" Nov 03 21:06:07.198586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.198565 5932 scope.go:122] "RemoveContainer" containerID="e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437" Nov 03 21:06:07.198904 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:07.198885 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437\": container with ID starting with e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437 not found: ID does not exist" containerID="e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437" Nov 03 21:06:07.198966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.198911 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437"} err="failed to get container status \"e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437\": rpc error: code = NotFound desc = could not find container \"e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437\": container with ID starting with e400d71d0c74f91906d1f4c11210ac56648039648c184d8f3cc8a1e088453437 not found: ID does not exist" Nov 03 21:06:07.347322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.347292 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm"] Nov 03 21:06:07.393381 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:07.393349 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d78b90-bc8b55f5f-67zrm"] Nov 03 21:06:08.052305 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:08.052272 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48102ead-fefd-47a6-9f72-4ecee9cc7bda" path="/var/lib/kubelet/pods/48102ead-fefd-47a6-9f72-4ecee9cc7bda/volumes" Nov 03 21:06:09.438472 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:09.438441 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz"] Nov 03 21:06:09.438892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:09.438667 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" containerID="cri-o://3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8" gracePeriod=30 Nov 03 21:06:14.152391 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:14.152350 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:06:14.725557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.725486 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:06:14.800679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800643 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.800845 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800690 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.800845 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800758 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.800845 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800821 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.801041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800884 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.801041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800936 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.801041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800935 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:14.801041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.800978 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.801245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801073 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.801245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801146 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:14.801245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801194 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket\") pod \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\" (UID: \"80d57f71-663f-4a44-a0c7-59db19c7ad1e\") " Nov 03 21:06:14.801245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801196 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:14.801469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801397 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:14.801523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801475 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:14.801885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801864 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.801993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801888 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.801993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801902 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.801993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801918 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.801993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.801931 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.803528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.803498 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "kube-api-access-7gsbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:14.803662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.803523 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:14.803662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.803580 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:14.803662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.803652 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token" pod "80d57f71-663f-4a44-a0c7-59db19c7ad1e" (UID: "80d57f71-663f-4a44-a0c7-59db19c7ad1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:14.903699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.903663 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.903699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.903699 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.903894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.903713 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:14.903894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:14.903728 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gsbn\" (UniqueName: \"kubernetes.io/projected/80d57f71-663f-4a44-a0c7-59db19c7ad1e-kube-api-access-7gsbn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:15.253034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.252995 5932 generic.go:356] "Generic (PLEG): container finished" podID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerID="3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8" exitCode=0 Nov 03 21:06:15.253445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.253060 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" Nov 03 21:06:15.253445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.253072 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" event={"ID":"80d57f71-663f-4a44-a0c7-59db19c7ad1e","Type":"ContainerDied","Data":"3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8"} Nov 03 21:06:15.253445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.253132 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz" event={"ID":"80d57f71-663f-4a44-a0c7-59db19c7ad1e","Type":"ContainerDied","Data":"d96abf8feb0f794db9e2bdc2ae19291cbe8406eb304c055e0ba3f0d85a73c13e"} Nov 03 21:06:15.253445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.253154 5932 scope.go:122] "RemoveContainer" containerID="3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8" Nov 03 21:06:15.265144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.265126 5932 scope.go:122] "RemoveContainer" containerID="0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47" Nov 03 21:06:15.274943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.274924 5932 scope.go:122] "RemoveContainer" containerID="3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8" Nov 03 21:06:15.275258 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:15.275234 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8\": container with ID starting with 3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8 not found: ID does not exist" containerID="3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8" Nov 03 21:06:15.275303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.275268 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8"} err="failed to get container status \"3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8\": rpc error: code = NotFound desc = could not find container \"3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8\": container with ID starting with 3e7afb7aeb3c3d89fc76ca328f33cd9735462acfb07bfe4f5449f521ce287cb8 not found: ID does not exist" Nov 03 21:06:15.275303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.275289 5932 scope.go:122] "RemoveContainer" containerID="0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47" Nov 03 21:06:15.275566 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:15.275546 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47\": container with ID starting with 0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47 not found: ID does not exist" containerID="0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47" Nov 03 21:06:15.275662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.275573 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47"} err="failed to get container status \"0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47\": rpc error: code = NotFound desc = could not find container \"0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47\": container with ID starting with 0bfa3431e52054741fe60ad5c9744b08cd45c01ec6db9e4eafb4a09700fc3b47 not found: ID does not exist" Nov 03 21:06:15.413215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.413182 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz"] Nov 03 21:06:15.459582 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:15.459543 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-37c5b0-openshift-default-5b6c87994f-ztqgz"] Nov 03 21:06:16.053233 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:16.053198 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" path="/var/lib/kubelet/pods/80d57f71-663f-4a44-a0c7-59db19c7ad1e/volumes" Nov 03 21:06:18.747115 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:18.747083 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk"] Nov 03 21:06:18.747673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:18.747273 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" containerID="cri-o://d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968" gracePeriod=30 Nov 03 21:06:19.645971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.645931 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg"] Nov 03 21:06:19.646637 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646582 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" Nov 03 21:06:19.646637 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646607 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" Nov 03 21:06:19.646637 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646637 5932 cpu_manager.go:414] "containerMap: removing container" podUID="48102ead-fefd-47a6-9f72-4ecee9cc7bda" containerName="payload-processing" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646642 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48102ead-fefd-47a6-9f72-4ecee9cc7bda" containerName="payload-processing" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646666 5932 cpu_manager.go:414] "containerMap: removing container" podUID="bf87bd98-07c6-49fc-8880-faeb56ed372c" containerName="revoke-keys" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646672 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf87bd98-07c6-49fc-8880-faeb56ed372c" containerName="revoke-keys" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646680 5932 cpu_manager.go:414] "containerMap: removing container" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646686 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646695 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ce949650-dc23-4e4d-9865-cd44ab218709" containerName="revoke-keys" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646700 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce949650-dc23-4e4d-9865-cd44ab218709" containerName="revoke-keys" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646710 5932 cpu_manager.go:414] "containerMap: removing container" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646714 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646724 5932 cpu_manager.go:414] "containerMap: removing container" podUID="152dd512-4484-489e-b01b-53ca4d512db6" containerName="payload-processing" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646729 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="152dd512-4484-489e-b01b-53ca4d512db6" containerName="payload-processing" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646741 5932 cpu_manager.go:414] "containerMap: removing container" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" Nov 03 21:06:19.646768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646746 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646852 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646860 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="48102ead-fefd-47a6-9f72-4ecee9cc7bda" containerName="payload-processing" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646869 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf87bd98-07c6-49fc-8880-faeb56ed372c" containerName="revoke-keys" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646876 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="82786355-cc14-4d9f-9912-e1c13d7e3b62" containerName="istio-proxy" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646882 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce949650-dc23-4e4d-9865-cd44ab218709" containerName="revoke-keys" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646888 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646893 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="152dd512-4484-489e-b01b-53ca4d512db6" containerName="payload-processing" Nov 03 21:06:19.647181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.646899 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="80d57f71-663f-4a44-a0c7-59db19c7ad1e" containerName="istio-proxy" Nov 03 21:06:19.650724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.650703 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.745483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.745454 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:06:19.745706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.745499 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:06:19.749693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.749671 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg"] Nov 03 21:06:19.765178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.765151 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.765379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.765355 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc7m8\" (UniqueName: \"kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.765448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.765390 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.866578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.866491 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc7m8\" (UniqueName: \"kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.866578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.866539 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.866797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.866646 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.867377 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.867357 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.868978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.868948 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.923974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.923941 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc7m8\" (UniqueName: \"kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8\") pod \"maas-api-revoke-sub-f314596c-tswrg\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:19.964743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:19.964709 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:20.350716 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:06:20.350674 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43350b94_433a_4a6d_a3bf_989f14f5515e.slice/crio-1584b0ecab535ff2dc52b114d7533b26377f70f21288d177f994a8f09692ddc6 WatchSource:0}: Error finding container 1584b0ecab535ff2dc52b114d7533b26377f70f21288d177f994a8f09692ddc6: Status 404 returned error can't find the container with id 1584b0ecab535ff2dc52b114d7533b26377f70f21288d177f994a8f09692ddc6 Nov 03 21:06:20.352468 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:20.352442 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg"] Nov 03 21:06:21.303705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:21.303659 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" event={"ID":"43350b94-433a-4a6d-a3bf-989f14f5515e","Type":"ContainerStarted","Data":"2c9f18bc0c6aa9d51aafac74985a2c70f46c4eb71271405ffe7b6a8aaf2688d4"} Nov 03 21:06:21.304175 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:21.303730 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" event={"ID":"43350b94-433a-4a6d-a3bf-989f14f5515e","Type":"ContainerStarted","Data":"1584b0ecab535ff2dc52b114d7533b26377f70f21288d177f994a8f09692ddc6"} Nov 03 21:06:21.414710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:21.414655 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" podStartSLOduration=2.414640335 podStartE2EDuration="2.414640335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:21.411363003 +0000 UTC m=+2189.476604490" watchObservedRunningTime="2026-11-03 21:06:21.414640335 +0000 UTC m=+2189.479881802" Nov 03 21:06:22.313096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:22.313056 5932 generic.go:356] "Generic (PLEG): container finished" podID="43350b94-433a-4a6d-a3bf-989f14f5515e" containerID="2c9f18bc0c6aa9d51aafac74985a2c70f46c4eb71271405ffe7b6a8aaf2688d4" exitCode=0 Nov 03 21:06:22.313564 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:22.313128 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" event={"ID":"43350b94-433a-4a6d-a3bf-989f14f5515e","Type":"ContainerDied","Data":"2c9f18bc0c6aa9d51aafac74985a2c70f46c4eb71271405ffe7b6a8aaf2688d4"} Nov 03 21:06:23.507306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.507276 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:23.622950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.622866 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca\") pod \"43350b94-433a-4a6d-a3bf-989f14f5515e\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " Nov 03 21:06:23.623136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.623119 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8\" (UniqueName: \"kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8\") pod \"43350b94-433a-4a6d-a3bf-989f14f5515e\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " Nov 03 21:06:23.623216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.623169 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca\") pod \"43350b94-433a-4a6d-a3bf-989f14f5515e\" (UID: \"43350b94-433a-4a6d-a3bf-989f14f5515e\") " Nov 03 21:06:23.623606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.623571 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca" pod "43350b94-433a-4a6d-a3bf-989f14f5515e" (UID: "43350b94-433a-4a6d-a3bf-989f14f5515e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:23.623952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.623877 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:23.626814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.626768 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca" pod "43350b94-433a-4a6d-a3bf-989f14f5515e" (UID: "43350b94-433a-4a6d-a3bf-989f14f5515e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:23.626814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.626793 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8" pod "43350b94-433a-4a6d-a3bf-989f14f5515e" (UID: "43350b94-433a-4a6d-a3bf-989f14f5515e"). InnerVolumeSpecName "kube-api-access-jc7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:23.724935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.724899 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43350b94-433a-4a6d-a3bf-989f14f5515e-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:23.724935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.724937 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc7m8\" (UniqueName: \"kubernetes.io/projected/43350b94-433a-4a6d-a3bf-989f14f5515e-kube-api-access-jc7m8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:23.994170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:23.994145 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 21:06:24.130112 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130075 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130123 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130165 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130200 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130256 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130325 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130472 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130596 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.130742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.130705 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs\") pod \"17462258-c58b-4211-8cdb-20283a6f1d61\" (UID: \"17462258-c58b-4211-8cdb-20283a6f1d61\") " Nov 03 21:06:24.131441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.131408 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:24.131656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.131601 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:24.132488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.132184 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:24.132813 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.132785 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:24.133146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.133121 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:24.134079 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.134058 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:24.134556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.134524 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:24.134934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.134908 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:24.135133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.135112 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb" pod "17462258-c58b-4211-8cdb-20283a6f1d61" (UID: "17462258-c58b-4211-8cdb-20283a6f1d61"). InnerVolumeSpecName "kube-api-access-hstqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:24.233090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.232996 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233031 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233043 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233056 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hstqb\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-kube-api-access-hstqb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233070 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17462258-c58b-4211-8cdb-20283a6f1d61-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233085 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233097 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17462258-c58b-4211-8cdb-20283a6f1d61-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233110 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17462258-c58b-4211-8cdb-20283a6f1d61-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.233418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.233123 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17462258-c58b-4211-8cdb-20283a6f1d61-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:24.329931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.329904 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" Nov 03 21:06:24.330116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.329941 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg" event={"ID":"43350b94-433a-4a6d-a3bf-989f14f5515e","Type":"ContainerDied","Data":"1584b0ecab535ff2dc52b114d7533b26377f70f21288d177f994a8f09692ddc6"} Nov 03 21:06:24.330116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.329997 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1584b0ecab535ff2dc52b114d7533b26377f70f21288d177f994a8f09692ddc6" Nov 03 21:06:24.331504 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.331475 5932 generic.go:356] "Generic (PLEG): container finished" podID="17462258-c58b-4211-8cdb-20283a6f1d61" containerID="d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968" exitCode=0 Nov 03 21:06:24.331608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.331524 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" Nov 03 21:06:24.331608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.331570 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" event={"ID":"17462258-c58b-4211-8cdb-20283a6f1d61","Type":"ContainerDied","Data":"d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968"} Nov 03 21:06:24.331714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.331637 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk" event={"ID":"17462258-c58b-4211-8cdb-20283a6f1d61","Type":"ContainerDied","Data":"165aed23f0b5fbbadce14ecd21ac33dfdf7578f4d2345fcc8d837e0e3774eb70"} Nov 03 21:06:24.331714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.331667 5932 scope.go:122] "RemoveContainer" containerID="d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968" Nov 03 21:06:24.344979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.344961 5932 scope.go:122] "RemoveContainer" containerID="d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968" Nov 03 21:06:24.345286 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:24.345264 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968\": container with ID starting with d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968 not found: ID does not exist" containerID="d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968" Nov 03 21:06:24.345353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.345294 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968"} err="failed to get container status \"d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968\": rpc error: code = NotFound desc = could not find container \"d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968\": container with ID starting with d1c3c79aa04fe159ccb65253852ca4a3d6a6f4c7dad2a557e4b372cd4662f968 not found: ID does not exist" Nov 03 21:06:24.488446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.488418 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk"] Nov 03 21:06:24.499712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:24.499686 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-69rmk"] Nov 03 21:06:26.053223 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:26.053186 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" path="/var/lib/kubelet/pods/17462258-c58b-4211-8cdb-20283a6f1d61/volumes" Nov 03 21:06:27.602142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602105 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6"] Nov 03 21:06:27.602841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602825 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" Nov 03 21:06:27.602889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602842 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" Nov 03 21:06:27.602889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602864 5932 cpu_manager.go:414] "containerMap: removing container" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" Nov 03 21:06:27.602889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602870 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" Nov 03 21:06:27.602889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602884 5932 cpu_manager.go:414] "containerMap: removing container" podUID="43350b94-433a-4a6d-a3bf-989f14f5515e" containerName="revoke-keys" Nov 03 21:06:27.602889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602889 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43350b94-433a-4a6d-a3bf-989f14f5515e" containerName="revoke-keys" Nov 03 21:06:27.603056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602981 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="17462258-c58b-4211-8cdb-20283a6f1d61" containerName="istio-proxy" Nov 03 21:06:27.603056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.602992 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="43350b94-433a-4a6d-a3bf-989f14f5515e" containerName="revoke-keys" Nov 03 21:06:27.603056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.603004 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcdb585-bf18-4dd5-a2dd-45a14c591520" containerName="istio-proxy" Nov 03 21:06:27.608265 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.608243 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.657229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.657196 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-df71d7-serving-cert\"" Nov 03 21:06:27.709150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.709114 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6"] Nov 03 21:06:27.782883 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.782850 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.783054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.783022 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.783099 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.783075 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.783179 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.783164 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fp49\" (UniqueName: \"kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.884445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.884408 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.884662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.884470 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.884662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.884513 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fp49\" (UniqueName: \"kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.884662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.884598 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.885293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.885272 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.886925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.886903 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.887027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.886921 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:27.942744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:27.942710 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fp49\" (UniqueName: \"kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49\") pod \"maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:28.222161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:28.222055 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:28.604364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:28.604282 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6"] Nov 03 21:06:28.609368 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:06:28.609341 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1039978d_115e_4129_9f85_2a09f7afd1a6.slice/crio-dbab3238a74e39a432570cdb1541fc1cdda1fc431c1093076f49971336858bba WatchSource:0}: Error finding container dbab3238a74e39a432570cdb1541fc1cdda1fc431c1093076f49971336858bba: Status 404 returned error can't find the container with id dbab3238a74e39a432570cdb1541fc1cdda1fc431c1093076f49971336858bba Nov 03 21:06:28.610863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:28.610845 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:06:29.375696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:29.375657 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" event={"ID":"1039978d-115e-4129-9f85-2a09f7afd1a6","Type":"ContainerStarted","Data":"dbab3238a74e39a432570cdb1541fc1cdda1fc431c1093076f49971336858bba"} Nov 03 21:06:30.385699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:30.385661 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" event={"ID":"1039978d-115e-4129-9f85-2a09f7afd1a6","Type":"ContainerStarted","Data":"0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26"} Nov 03 21:06:30.386117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:30.385896 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:30.496996 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:30.496945 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" podStartSLOduration=2.5133441359999997 podStartE2EDuration="3.496930794s" totalImagesPullingTime="983.586658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:30.494997692 +0000 UTC m=+2198.560239166" watchObservedRunningTime="2026-11-03 21:06:30.496930794 +0000 UTC m=+2198.562172263" Nov 03 21:06:32.206568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.206529 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv"] Nov 03 21:06:32.207018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.206767 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" podUID="adf97a22-3157-4516-9cdf-de395e9b0faf" containerName="maas-api" containerID="cri-o://f542f8d35912652384ad272401e626ff2a3c75de7e7f19a724b4eafc9a0cc599" gracePeriod=30 Nov 03 21:06:32.483863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.482558 5932 generic.go:356] "Generic (PLEG): container finished" podID="adf97a22-3157-4516-9cdf-de395e9b0faf" containerID="f542f8d35912652384ad272401e626ff2a3c75de7e7f19a724b4eafc9a0cc599" exitCode=0 Nov 03 21:06:32.484684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.484596 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" event={"ID":"adf97a22-3157-4516-9cdf-de395e9b0faf","Type":"ContainerDied","Data":"f542f8d35912652384ad272401e626ff2a3c75de7e7f19a724b4eafc9a0cc599"} Nov 03 21:06:32.535046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.535016 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:06:32.681642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.677910 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs\") pod \"adf97a22-3157-4516-9cdf-de395e9b0faf\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " Nov 03 21:06:32.681642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.678099 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx\" (UniqueName: \"kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx\") pod \"adf97a22-3157-4516-9cdf-de395e9b0faf\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " Nov 03 21:06:32.681642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.678136 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\") pod \"adf97a22-3157-4516-9cdf-de395e9b0faf\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " Nov 03 21:06:32.681642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.678413 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca\") pod \"adf97a22-3157-4516-9cdf-de395e9b0faf\" (UID: \"adf97a22-3157-4516-9cdf-de395e9b0faf\") " Nov 03 21:06:32.681642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.679441 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca" pod "adf97a22-3157-4516-9cdf-de395e9b0faf" (UID: "adf97a22-3157-4516-9cdf-de395e9b0faf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:32.692668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.690420 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs" pod "adf97a22-3157-4516-9cdf-de395e9b0faf" (UID: "adf97a22-3157-4516-9cdf-de395e9b0faf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:32.692668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.690688 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx" pod "adf97a22-3157-4516-9cdf-de395e9b0faf" (UID: "adf97a22-3157-4516-9cdf-de395e9b0faf"). InnerVolumeSpecName "kube-api-access-vd4dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:32.693857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.693829 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls" pod "adf97a22-3157-4516-9cdf-de395e9b0faf" (UID: "adf97a22-3157-4516-9cdf-de395e9b0faf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:06:32.793095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.790257 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf97a22-3157-4516-9cdf-de395e9b0faf-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:32.793095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.790294 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:32.793095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.790310 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd4dx\" (UniqueName: \"kubernetes.io/projected/adf97a22-3157-4516-9cdf-de395e9b0faf-kube-api-access-vd4dx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:32.793095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:32.790324 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf97a22-3157-4516-9cdf-de395e9b0faf-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:33.493617 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:33.493586 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" Nov 03 21:06:33.494056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:33.493596 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv" event={"ID":"adf97a22-3157-4516-9cdf-de395e9b0faf","Type":"ContainerDied","Data":"89c2567053ab8b4326f96d6b0fd082a3925d4d47c8f4af4648ae949f6b43721e"} Nov 03 21:06:33.494056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:33.493713 5932 scope.go:122] "RemoveContainer" containerID="f542f8d35912652384ad272401e626ff2a3c75de7e7f19a724b4eafc9a0cc599" Nov 03 21:06:33.651818 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:33.651780 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv"] Nov 03 21:06:33.697191 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:33.697155 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-878695-6b7458d845-2flmv"] Nov 03 21:06:34.057551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:34.057512 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf97a22-3157-4516-9cdf-de395e9b0faf" path="/var/lib/kubelet/pods/adf97a22-3157-4516-9cdf-de395e9b0faf/volumes" Nov 03 21:06:34.517073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:34.517039 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx"] Nov 03 21:06:34.562671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:34.562607 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-878695-pfrjx"] Nov 03 21:06:35.187662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.187613 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57"] Nov 03 21:06:35.188606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.188584 5932 cpu_manager.go:414] "containerMap: removing container" podUID="adf97a22-3157-4516-9cdf-de395e9b0faf" containerName="maas-api" Nov 03 21:06:35.188695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.188610 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf97a22-3157-4516-9cdf-de395e9b0faf" containerName="maas-api" Nov 03 21:06:35.188843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.188830 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf97a22-3157-4516-9cdf-de395e9b0faf" containerName="maas-api" Nov 03 21:06:35.193799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.193774 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.247354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.247320 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57"] Nov 03 21:06:35.254001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.253974 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e68f8112-serving-cert\"" Nov 03 21:06:35.263804 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.263773 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.264009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.263900 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfcg\" (UniqueName: \"kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.264009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.263966 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.264009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.263991 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.365074 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.365037 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.365246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.365137 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfcg\" (UniqueName: \"kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.365246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.365198 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.365246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.365227 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.365376 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:35.365358 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-serving-cert: secret "maas-api-e2e-mt-e68f8112-serving-cert" not found Nov 03 21:06:35.365440 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:35.365429 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls podName:3e1d8531-44c6-4f83-a161-2be10803677b nodeName:}" failed. No retries permitted until 2026-11-03 21:06:35.865407804 +0000 UTC m=+2203.930649269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls") pod "maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" (UID: "3e1d8531-44c6-4f83-a161-2be10803677b") : secret "maas-api-e2e-mt-e68f8112-serving-cert" not found Nov 03 21:06:35.365741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.365718 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.367780 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.367742 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.424775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.424736 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfcg\" (UniqueName: \"kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.860679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.860647 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25"] Nov 03 21:06:35.864952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.864926 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ba7e83-wks25"] Nov 03 21:06:35.877502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.877474 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:35.880064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:35.880042 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\") pod \"maas-api-e2e-mt-e68f8112-7598458dbc-sfq57\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:36.053523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:36.053485 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6" path="/var/lib/kubelet/pods/34272f6f-e3ef-45ef-9ce0-aa5a0d4940a6/volumes" Nov 03 21:06:36.054064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:36.054046 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bcb305-a8b0-421e-b79c-5f7929f3b5dc" path="/var/lib/kubelet/pods/b9bcb305-a8b0-421e-b79c-5f7929f3b5dc/volumes" Nov 03 21:06:36.108157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:36.108125 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:36.399315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:36.399285 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:06:36.563357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:36.563326 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57"] Nov 03 21:06:37.533241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:37.533205 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" event={"ID":"3e1d8531-44c6-4f83-a161-2be10803677b","Type":"ContainerStarted","Data":"a339246a77caaff3a61bf59d8c17b2ac6b7d465db776b1ffa72c52897d7d099b"} Nov 03 21:06:38.542080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:38.542044 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" event={"ID":"3e1d8531-44c6-4f83-a161-2be10803677b","Type":"ContainerStarted","Data":"94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0"} Nov 03 21:06:38.542510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:38.542201 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:38.686645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:38.681666 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" podStartSLOduration=2.682143042 podStartE2EDuration="3.681536034s" totalImagesPullingTime="999.392992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:38.657071204 +0000 UTC m=+2206.722312678" watchObservedRunningTime="2026-11-03 21:06:38.681536034 +0000 UTC m=+2206.746777508" Nov 03 21:06:42.657399 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:42.657364 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw"] Nov 03 21:06:42.657853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:42.657581 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" containerID="cri-o://be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65" gracePeriod=30 Nov 03 21:06:44.212895 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:44.212844 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:06:44.555855 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:44.555783 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:06:47.929982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:47.929922 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 21:06:48.064319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064280 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.064507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064356 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.064507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064393 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.064723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064706 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.064776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064709 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:48.064832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064776 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.064885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064838 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.064885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064863 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:48.065000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064896 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.065000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064881 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:48.065000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064926 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.065000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.064954 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy\") pod \"af9e6dfb-9c04-4894-808f-d8364790e47b\" (UID: \"af9e6dfb-9c04-4894-808f-d8364790e47b\") " Nov 03 21:06:48.065222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065097 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:48.065222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065131 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:06:48.065470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065449 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af9e6dfb-9c04-4894-808f-d8364790e47b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.065556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065474 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.065556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065489 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.065556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065502 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.065556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.065515 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.067323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.067293 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:06:48.067454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.067368 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:06:48.067522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.067473 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "kube-api-access-w7gsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:48.067581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.067522 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token" pod "af9e6dfb-9c04-4894-808f-d8364790e47b" (UID: "af9e6dfb-9c04-4894-808f-d8364790e47b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:06:48.166524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.166487 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.166524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.166519 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gsl\" (UniqueName: \"kubernetes.io/projected/af9e6dfb-9c04-4894-808f-d8364790e47b-kube-api-access-w7gsl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.166524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.166529 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.166813 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.166538 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af9e6dfb-9c04-4894-808f-d8364790e47b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:06:48.623063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.623021 5932 generic.go:356] "Generic (PLEG): container finished" podID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerID="be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65" exitCode=0 Nov 03 21:06:48.623242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.623095 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" Nov 03 21:06:48.623242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.623113 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" event={"ID":"af9e6dfb-9c04-4894-808f-d8364790e47b","Type":"ContainerDied","Data":"be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65"} Nov 03 21:06:48.623242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.623184 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" event={"ID":"af9e6dfb-9c04-4894-808f-d8364790e47b","Type":"ContainerDied","Data":"370d88ba98e678c5777d8586a1a07cc667731f2f8951c8dbb1cf3429f8bd2581"} Nov 03 21:06:48.623242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.623213 5932 scope.go:122] "RemoveContainer" containerID="be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65" Nov 03 21:06:48.635387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.635368 5932 scope.go:122] "RemoveContainer" containerID="9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b" Nov 03 21:06:48.644936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.644915 5932 scope.go:122] "RemoveContainer" containerID="be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65" Nov 03 21:06:48.645230 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:48.645208 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65\": container with ID starting with be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65 not found: ID does not exist" containerID="be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65" Nov 03 21:06:48.645290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.645239 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65"} err="failed to get container status \"be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65\": rpc error: code = NotFound desc = could not find container \"be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65\": container with ID starting with be7984058b4be3a8d1d3003c34195aca996e1180fcc47210cca2f03c96598e65 not found: ID does not exist" Nov 03 21:06:48.645290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.645257 5932 scope.go:122] "RemoveContainer" containerID="9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b" Nov 03 21:06:48.645482 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:06:48.645462 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b\": container with ID starting with 9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b not found: ID does not exist" containerID="9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b" Nov 03 21:06:48.645540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.645491 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b"} err="failed to get container status \"9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b\": rpc error: code = NotFound desc = could not find container \"9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b\": container with ID starting with 9a196c41d4a818a7beea7aedbe0014059476aeba7cb46c5e2c9b3137b944163b not found: ID does not exist" Nov 03 21:06:48.827994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.827960 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw"] Nov 03 21:06:48.834854 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.834829 5932 patch_prober.go:29] interesting pod/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:06:48.834947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.834899 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:48.874182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:48.874101 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ba7e83-openshift-default-9bfc6b8b5-2clqw"] Nov 03 21:06:50.052824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:50.052794 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" path="/var/lib/kubelet/pods/af9e6dfb-9c04-4894-808f-d8364790e47b/volumes" Nov 03 21:06:52.016677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.016606 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569"] Nov 03 21:06:52.017355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.017340 5932 cpu_manager.go:414] "containerMap: removing container" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" Nov 03 21:06:52.017402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.017357 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" Nov 03 21:06:52.017508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.017497 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" Nov 03 21:06:52.017544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.017513 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" Nov 03 21:06:52.022736 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.022716 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.062989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.062949 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6"] Nov 03 21:06:52.063999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.063972 5932 cpu_manager.go:414] "containerMap: removing container" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" Nov 03 21:06:52.063999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.063996 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af9e6dfb-9c04-4894-808f-d8364790e47b" containerName="istio-proxy" Nov 03 21:06:52.068593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.068566 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb"] Nov 03 21:06:52.068749 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.068708 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.073870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.073844 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.081520 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.081493 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb"] Nov 03 21:06:52.086260 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.086239 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.105607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.105582 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\"" Nov 03 21:06:52.115025 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.114994 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115077 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115106 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115168 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569"] Nov 03 21:06:52.115318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115256 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpb4s\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115303 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115358 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115440 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115497 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.115575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.115533 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.187643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.187585 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6"] Nov 03 21:06:52.192814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.192779 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb"] Nov 03 21:06:52.217364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217321 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.217559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217368 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.217559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217435 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.217896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217870 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.217992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217903 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.217992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217936 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.217990 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218076 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218128 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218238 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218285 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218336 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.218418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218371 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218408 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.218601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218449 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218455 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218477 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.218831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218640 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.218831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218659 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.218831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218691 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxm7m\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.218831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218709 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzg7t\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.218831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218769 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.218831 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218804 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.219177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.218835 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.219177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219024 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.219177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219138 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.219350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219189 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.219350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219215 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.219350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219249 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.219350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219304 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.219350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219334 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.219647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219354 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.219647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219447 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.219647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219471 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.219647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219502 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.219647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219563 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpb4s\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.219647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219586 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.219980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219654 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.219980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219739 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.219980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219784 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.219980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.219858 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lp58\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.221443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.221422 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.221505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.221477 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.288019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.287942 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb"] Nov 03 21:06:52.290266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.289725 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.303977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.303943 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpb4s\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rz569\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.320991 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.320943 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321031 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321062 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321091 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321349 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321228 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.321349 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321298 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.321349 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321329 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321373 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.321517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321383 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321395 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321403 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.321517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321462 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.321517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321500 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321526 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321561 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lp58\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321616 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321688 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321697 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321756 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321789 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.321871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321829 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321873 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321909 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321634 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321958 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.321999 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322040 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322069 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxm7m\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322100 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzg7t\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322137 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322160 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322193 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322222 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.322320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322289 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.323077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322452 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.323077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322468 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.323077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.322542 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.323268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.323212 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.323371 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.323329 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.323371 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.323353 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.323682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.323610 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.324786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.323877 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.324786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.324360 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.324786 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.324501 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.325207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.325184 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.328162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.328135 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.328309 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.328290 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.329615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.329590 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.380967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.380923 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.381162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.381115 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.381229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.381178 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.381229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.381204 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lp58\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.385841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.385809 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\"" Nov 03 21:06:52.387019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.386996 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxm7m\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-4xld6\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.387441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.387411 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:52.388072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.388055 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:52.389245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.389225 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzg7t\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.396671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.396645 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:52.403566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.403539 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:52.801219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.801188 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6"] Nov 03 21:06:52.803893 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:06:52.803861 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e55545e_c3ca_4550_86f8_ba0b7b12c513.slice/crio-8516c3c77194e067673a61db7241e2ba00dfb8370075a661a0d3f775ed998f60 WatchSource:0}: Error finding container 8516c3c77194e067673a61db7241e2ba00dfb8370075a661a0d3f775ed998f60: Status 404 returned error can't find the container with id 8516c3c77194e067673a61db7241e2ba00dfb8370075a661a0d3f775ed998f60 Nov 03 21:06:52.806936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.806898 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.807029 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.806978 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.807029 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.807016 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.848449 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.848409 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569"] Nov 03 21:06:52.852202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.852173 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb"] Nov 03 21:06:52.854348 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:06:52.854312 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1798e6_5adf_4c58_9409_869c2ffd2d38.slice/crio-b9b992e2c6203164c926ba589387cafe271ff50b243a9b91ab32f6e74cd9d6f9 WatchSource:0}: Error finding container b9b992e2c6203164c926ba589387cafe271ff50b243a9b91ab32f6e74cd9d6f9: Status 404 returned error can't find the container with id b9b992e2c6203164c926ba589387cafe271ff50b243a9b91ab32f6e74cd9d6f9 Nov 03 21:06:52.857006 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:06:52.856977 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c454362_f7e1_4ef6_94dd_fb29158f12b9.slice/crio-26d51fc019d709e4bc1a5d2ddee24a448797a84958eac1f92dab52accfa74bc4 WatchSource:0}: Error finding container 26d51fc019d709e4bc1a5d2ddee24a448797a84958eac1f92dab52accfa74bc4: Status 404 returned error can't find the container with id 26d51fc019d709e4bc1a5d2ddee24a448797a84958eac1f92dab52accfa74bc4 Nov 03 21:06:52.858097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.857948 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.858097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.858035 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.860088 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.860058 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.860203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.860131 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.860203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.860178 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.861234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.861204 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.894911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.894876 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb"] Nov 03 21:06:52.898073 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:06:52.898042 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393d8b8c_f4e6_4e66_b329_8cad52013719.slice/crio-fca1744288c9f27d9617984405542c1cf4631443bf5f3a8fa904b317386d3808 WatchSource:0}: Error finding container fca1744288c9f27d9617984405542c1cf4631443bf5f3a8fa904b317386d3808: Status 404 returned error can't find the container with id fca1744288c9f27d9617984405542c1cf4631443bf5f3a8fa904b317386d3808 Nov 03 21:06:52.901138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.901105 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.901225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.901172 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:52.901225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:52.901203 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:06:53.668259 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.668226 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" event={"ID":"1f1798e6-5adf-4c58-9409-869c2ffd2d38","Type":"ContainerStarted","Data":"40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017"} Nov 03 21:06:53.668754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.668724 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" event={"ID":"1f1798e6-5adf-4c58-9409-869c2ffd2d38","Type":"ContainerStarted","Data":"b9b992e2c6203164c926ba589387cafe271ff50b243a9b91ab32f6e74cd9d6f9"} Nov 03 21:06:53.670901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.670868 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" event={"ID":"8e55545e-c3ca-4550-86f8-ba0b7b12c513","Type":"ContainerStarted","Data":"26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3"} Nov 03 21:06:53.671031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.670923 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" event={"ID":"8e55545e-c3ca-4550-86f8-ba0b7b12c513","Type":"ContainerStarted","Data":"8516c3c77194e067673a61db7241e2ba00dfb8370075a661a0d3f775ed998f60"} Nov 03 21:06:53.673490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.673460 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" event={"ID":"393d8b8c-f4e6-4e66-b329-8cad52013719","Type":"ContainerStarted","Data":"493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887"} Nov 03 21:06:53.673729 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.673709 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" event={"ID":"393d8b8c-f4e6-4e66-b329-8cad52013719","Type":"ContainerStarted","Data":"fca1744288c9f27d9617984405542c1cf4631443bf5f3a8fa904b317386d3808"} Nov 03 21:06:53.676544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.676519 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" event={"ID":"4c454362-f7e1-4ef6-94dd-fb29158f12b9","Type":"ContainerStarted","Data":"7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08"} Nov 03 21:06:53.676771 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.676750 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" event={"ID":"4c454362-f7e1-4ef6-94dd-fb29158f12b9","Type":"ContainerStarted","Data":"26d51fc019d709e4bc1a5d2ddee24a448797a84958eac1f92dab52accfa74bc4"} Nov 03 21:06:53.808349 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.808285 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podStartSLOduration=2.8082619170000003 podStartE2EDuration="2.808261917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:53.789234736 +0000 UTC m=+2221.854476213" watchObservedRunningTime="2026-11-03 21:06:53.808261917 +0000 UTC m=+2221.873503387" Nov 03 21:06:53.954765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:53.954611 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podStartSLOduration=2.954580116 podStartE2EDuration="2.954580116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:53.936163434 +0000 UTC m=+2222.001404907" watchObservedRunningTime="2026-11-03 21:06:53.954580116 +0000 UTC m=+2222.019821599" Nov 03 21:06:54.135996 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.135884 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podStartSLOduration=2.135861626 podStartE2EDuration="2.135861626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:54.080742295 +0000 UTC m=+2222.145983768" watchObservedRunningTime="2026-11-03 21:06:54.135861626 +0000 UTC m=+2222.201103102" Nov 03 21:06:54.331122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.326480 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podStartSLOduration=3.326456559 podStartE2EDuration="3.326456559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:06:54.237304649 +0000 UTC m=+2222.302546123" watchObservedRunningTime="2026-11-03 21:06:54.326456559 +0000 UTC m=+2222.391698041" Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.429591 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.429683 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.429712 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.429740 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.440775 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.440864 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.441728 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:06:54.445646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.441809 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:06:54.465592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.465291 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:06:54.465592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.465392 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:06:54.465592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.465493 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:06:54.465592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.465519 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:06:54.692540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.692503 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6"] Nov 03 21:06:54.741643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.731130 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:06:54.808649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.792648 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh"] Nov 03 21:06:54.925645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:54.921731 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:55.408657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.357325 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/health\": context deadline exceeded" Nov 03 21:06:55.433740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.389460 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:06:55.434018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.433988 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:06:55.441719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.417940 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:06:55.457648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.453701 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:06:55.482748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.464935 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:06:55.493684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.482998 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:06:55.523745 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.464999 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:06:55.548650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.523993 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:06:55.679700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.678717 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:55.714802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.714742 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 03 21:06:55.732747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.732701 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:55.733405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.733375 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 21:06:55.733955 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.733935 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:06:55.740721 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.740685 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:55.787648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.785059 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:06:55.787648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.785523 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:55.809647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:55.805577 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:56.432656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.417924 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:06:56.432656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.417984 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:06:56.432656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.420451 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:06:56.432656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.420496 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:06:56.458678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.436267 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:06:56.458678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.436326 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:06:56.459177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.459146 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:06:56.459506 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.459467 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:06:56.734707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.733678 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 21:06:56.734707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.734206 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:06:56.760302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.760257 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:56.761004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.760978 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 21:06:56.789649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.770132 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:56.789649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.770538 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:06:56.791309 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.791276 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 21:06:56.798245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.798205 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:06:56.798904 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.798877 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:56.799402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.799368 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 21:06:56.799583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.799569 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 21:06:56.807062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.807030 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:06:56.807549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.807531 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:06:56.856802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.841659 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:06:56.898395 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:56.898366 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:06:57.015975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.015890 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:06:57.117649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.103719 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:57.117649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.104183 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 21:06:57.416712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.416674 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:06:57.416985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.416958 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:06:57.481653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.475210 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:06:57.481653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.475269 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:06:57.533655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.513381 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:06:57.598131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.598093 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:06:57.598313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.589299 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:06:57.598438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.598418 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:06:57.875706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.873333 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 21:06:57.898654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.896699 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:06:57.945644 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.903716 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 21:06:57.945644 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:57.929830 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.112647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.111661 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.410986 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.388784 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.430700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.419227 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:06:58.478286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.478243 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:06:58.478519 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.446174 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:06:58.478732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.478704 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:06:58.625695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.484548 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:06:58.625695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.623024 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:06:58.667048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.667010 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:06:58.907749 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.907704 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.911698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.874746 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 21:06:58.917178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.897299 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.928924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.898123 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 21:06:58.975647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.904032 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.975647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.907770 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:06:58.975647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.845097 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:58.975647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:58.930470 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:59.113284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.112714 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 03 21:06:59.450649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.399592 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:06:59.450649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.399669 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:06:59.450649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.399749 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:06:59.450649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.399769 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:06:59.450649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.400820 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:06:59.450649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.400858 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:06:59.462775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.462735 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:06:59.463019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.462990 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:06:59.725793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.725684 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" podUID="f2d1bece-24b9-4202-a5c6-285f3e99834a" containerName="payload-processing" containerID="cri-o://63bbc8518e6b12fc792b7bf0669d3ab7fd75ece7ed31adece03753d008b3423c" gracePeriod=25 Nov 03 21:06:59.809321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.809277 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" podUID="2c053419-7452-4329-8ff0-71eff6aa8c02" containerName="payload-pre-processing" containerID="cri-o://00d6605d6fd191b069e82a19e5235e52fd416fec7e4732f66dda50bcd78c6d1d" gracePeriod=25 Nov 03 21:06:59.891653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.890498 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:59.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.927393 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:06:59.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.927806 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 21:06:59.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.928086 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 21:06:59.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.928382 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:06:59.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.928441 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:06:59.940686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.939849 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:59.941879 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.941829 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:06:59.942348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.942300 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": context deadline exceeded" Nov 03 21:06:59.942888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.942848 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": context deadline exceeded" Nov 03 21:06:59.943412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:06:59.943360 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:07:00.185760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.159757 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:00.436756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.423287 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:07:00.436756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.423347 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:07:00.436756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.423426 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:07:00.436756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.423448 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:07:00.436756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.423775 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:07:00.436756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.423810 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:07:00.466452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.465768 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:07:00.466452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.465842 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:07:00.644653 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:00.635494 5932 logging.go:55] [core] [Channel #6803 SubChannel #6804]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.209:50052", ServerName: "10.132.0.209:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.209:50052: connect: connection refused" Nov 03 21:07:00.702207 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:00.700740 5932 logging.go:55] [core] [Channel #6805 SubChannel #6806]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.208:50052", ServerName: "10.132.0.208:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.208:50052: connect: connection refused" Nov 03 21:07:00.739140 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.726130 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:07:00.786113 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.786079 5932 generic.go:356] "Generic (PLEG): container finished" podID="2c053419-7452-4329-8ff0-71eff6aa8c02" containerID="00d6605d6fd191b069e82a19e5235e52fd416fec7e4732f66dda50bcd78c6d1d" exitCode=0 Nov 03 21:07:00.787231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.786719 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" event={"ID":"2c053419-7452-4329-8ff0-71eff6aa8c02","Type":"ContainerDied","Data":"00d6605d6fd191b069e82a19e5235e52fd416fec7e4732f66dda50bcd78c6d1d"} Nov 03 21:07:00.800711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.800678 5932 generic.go:356] "Generic (PLEG): container finished" podID="f2d1bece-24b9-4202-a5c6-285f3e99834a" containerID="63bbc8518e6b12fc792b7bf0669d3ab7fd75ece7ed31adece03753d008b3423c" exitCode=0 Nov 03 21:07:00.800887 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.800865 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" event={"ID":"f2d1bece-24b9-4202-a5c6-285f3e99834a","Type":"ContainerDied","Data":"63bbc8518e6b12fc792b7bf0669d3ab7fd75ece7ed31adece03753d008b3423c"} Nov 03 21:07:00.827151 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.827121 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:07:00.850645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.846898 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 21:07:00.897342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.897308 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 21:07:00.902892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.900960 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 21:07:00.904979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.904962 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 21:07:00.909276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.909261 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:07:00.943654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:00.943258 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:01.100516 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.100482 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 21:07:01.248446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.244739 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:07:01.360053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.359860 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:07:01.388559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.388509 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:07:01.388785 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.388572 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:07:01.388893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.388876 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:07:01.388959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.388910 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:07:01.397553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.397526 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:07:01.397830 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.397807 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.404841 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.404890 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.405537 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr\" (UniqueName: \"kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr\") pod \"2c053419-7452-4329-8ff0-71eff6aa8c02\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.406298 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume\") pod \"f2d1bece-24b9-4202-a5c6-285f3e99834a\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.407049 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume" pod "f2d1bece-24b9-4202-a5c6-285f3e99834a" (UID: "f2d1bece-24b9-4202-a5c6-285f3e99834a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.407137 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume\") pod \"2c053419-7452-4329-8ff0-71eff6aa8c02\" (UID: \"2c053419-7452-4329-8ff0-71eff6aa8c02\") " Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.407784 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume" pod "2c053419-7452-4329-8ff0-71eff6aa8c02" (UID: "2c053419-7452-4329-8ff0-71eff6aa8c02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.408521 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d1bece-24b9-4202-a5c6-285f3e99834a-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:01.408720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.408546 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c053419-7452-4329-8ff0-71eff6aa8c02-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:01.424223 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.424172 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr" pod "2c053419-7452-4329-8ff0-71eff6aa8c02" (UID: "2c053419-7452-4329-8ff0-71eff6aa8c02"). InnerVolumeSpecName "kube-api-access-nsmcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:01.511755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.511661 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k\" (UniqueName: \"kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k\") pod \"f2d1bece-24b9-4202-a5c6-285f3e99834a\" (UID: \"f2d1bece-24b9-4202-a5c6-285f3e99834a\") " Nov 03 21:07:01.512223 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.512200 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsmcr\" (UniqueName: \"kubernetes.io/projected/2c053419-7452-4329-8ff0-71eff6aa8c02-kube-api-access-nsmcr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:01.515392 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.515359 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k" pod "f2d1bece-24b9-4202-a5c6-285f3e99834a" (UID: "f2d1bece-24b9-4202-a5c6-285f3e99834a"). InnerVolumeSpecName "kube-api-access-xmq2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:01.618131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.617007 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmq2k\" (UniqueName: \"kubernetes.io/projected/f2d1bece-24b9-4202-a5c6-285f3e99834a-kube-api-access-xmq2k\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:01.627174 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.627127 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" podUID="f2d1bece-24b9-4202-a5c6-285f3e99834a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.209:50052\" within 1s: context deadline exceeded" Nov 03 21:07:01.668467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.668419 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" podUID="2c053419-7452-4329-8ff0-71eff6aa8c02" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.208:50052\" within 1s: context deadline exceeded" Nov 03 21:07:01.825702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.822134 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" event={"ID":"2c053419-7452-4329-8ff0-71eff6aa8c02","Type":"ContainerDied","Data":"dc9f7e70f5d7c5656bb1719166f2d257cb8a827c579ccae64767034dcc2facf3"} Nov 03 21:07:01.825702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.822213 5932 scope.go:122] "RemoveContainer" containerID="00d6605d6fd191b069e82a19e5235e52fd416fec7e4732f66dda50bcd78c6d1d" Nov 03 21:07:01.825702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.822366 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh" Nov 03 21:07:01.832517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.832102 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" event={"ID":"f2d1bece-24b9-4202-a5c6-285f3e99834a","Type":"ContainerDied","Data":"e409b58a591b260e13da1df9e3c65a20107a8829a6ff9e95978e4bdc3516b93c"} Nov 03 21:07:01.832517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.832208 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6" Nov 03 21:07:01.915487 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:01.915155 5932 scope.go:122] "RemoveContainer" containerID="63bbc8518e6b12fc792b7bf0669d3ab7fd75ece7ed31adece03753d008b3423c" Nov 03 21:07:02.089808 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.089706 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh"] Nov 03 21:07:02.095331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.095295 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0776c6-77c94cdb47-9ndmh"] Nov 03 21:07:02.113468 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.113428 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg"] Nov 03 21:07:02.113713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.113688 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" podUID="6a22bcae-cd8d-4522-bf24-0392d534bfc9" containerName="maas-api" containerID="cri-o://e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80" gracePeriod=30 Nov 03 21:07:02.197727 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.197691 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6"] Nov 03 21:07:02.233975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.233448 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0776c6-6589f5d55-knkl6"] Nov 03 21:07:02.446156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.445707 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 21:07:02.446156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.445787 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 21:07:02.451182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.451149 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 21:07:02.451545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.451499 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 21:07:02.451826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.451806 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:07:02.451984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.451962 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:07:02.452172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.452155 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Nov 03 21:07:02.452292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.452273 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Nov 03 21:07:02.686854 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.686819 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:07:02.801575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.800406 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs\") pod \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " Nov 03 21:07:02.801575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.800481 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca\") pod \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " Nov 03 21:07:02.801575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.800550 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629\" (UniqueName: \"kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629\") pod \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " Nov 03 21:07:02.801575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.800717 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\") pod \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\" (UID: \"6a22bcae-cd8d-4522-bf24-0392d534bfc9\") " Nov 03 21:07:02.802475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.802425 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca" pod "6a22bcae-cd8d-4522-bf24-0392d534bfc9" (UID: "6a22bcae-cd8d-4522-bf24-0392d534bfc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:02.804943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.804915 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls" pod "6a22bcae-cd8d-4522-bf24-0392d534bfc9" (UID: "6a22bcae-cd8d-4522-bf24-0392d534bfc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:02.809171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.808729 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629" pod "6a22bcae-cd8d-4522-bf24-0392d534bfc9" (UID: "6a22bcae-cd8d-4522-bf24-0392d534bfc9"). InnerVolumeSpecName "kube-api-access-bl629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:02.820822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.820773 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs" pod "6a22bcae-cd8d-4522-bf24-0392d534bfc9" (UID: "6a22bcae-cd8d-4522-bf24-0392d534bfc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:02.861020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.855488 5932 generic.go:356] "Generic (PLEG): container finished" podID="6a22bcae-cd8d-4522-bf24-0392d534bfc9" containerID="e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80" exitCode=0 Nov 03 21:07:02.861020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.855534 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" event={"ID":"6a22bcae-cd8d-4522-bf24-0392d534bfc9","Type":"ContainerDied","Data":"e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80"} Nov 03 21:07:02.861020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.855636 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" event={"ID":"6a22bcae-cd8d-4522-bf24-0392d534bfc9","Type":"ContainerDied","Data":"f503f9f3e8f7d8402b9e982df243cf5ae7a8edeb8350f47ac37bc63b254e7818"} Nov 03 21:07:02.861020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.855674 5932 scope.go:122] "RemoveContainer" containerID="e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80" Nov 03 21:07:02.861020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.855834 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg" Nov 03 21:07:02.887952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.887926 5932 scope.go:122] "RemoveContainer" containerID="e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80" Nov 03 21:07:02.888428 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:02.888398 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80\": container with ID starting with e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80 not found: ID does not exist" containerID="e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80" Nov 03 21:07:02.888506 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.888455 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80"} err="failed to get container status \"e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80\": rpc error: code = NotFound desc = could not find container \"e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80\": container with ID starting with e003e09c8ea0031f9503cf77818b8c651c177d412484c939d3e7f84c0e21aa80 not found: ID does not exist" Nov 03 21:07:02.903734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.903663 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:02.903734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.903699 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a22bcae-cd8d-4522-bf24-0392d534bfc9-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:02.903734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.903729 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a22bcae-cd8d-4522-bf24-0392d534bfc9-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:02.904009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:02.903748 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl629\" (UniqueName: \"kubernetes.io/projected/6a22bcae-cd8d-4522-bf24-0392d534bfc9-kube-api-access-bl629\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:03.033414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:03.033362 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg"] Nov 03 21:07:03.039348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:03.039311 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d78b90-7b4cf47845-kvrfg"] Nov 03 21:07:03.389711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:03.389300 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rz569 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Nov 03 21:07:03.389711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:03.389367 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Nov 03 21:07:04.127648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.125254 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c053419-7452-4329-8ff0-71eff6aa8c02" path="/var/lib/kubelet/pods/2c053419-7452-4329-8ff0-71eff6aa8c02/volumes" Nov 03 21:07:04.158648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.147377 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a22bcae-cd8d-4522-bf24-0392d534bfc9" path="/var/lib/kubelet/pods/6a22bcae-cd8d-4522-bf24-0392d534bfc9/volumes" Nov 03 21:07:04.158648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.148018 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d1bece-24b9-4202-a5c6-285f3e99834a" path="/var/lib/kubelet/pods/f2d1bece-24b9-4202-a5c6-285f3e99834a/volumes" Nov 03 21:07:04.432300 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.432197 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:04.432947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.432896 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:04.446767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.446714 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:04.447018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.446975 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:04.460126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.460006 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:07:04.460452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.460417 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:04.721740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.721364 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:07:04.723357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.722705 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:07:04.727282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.727255 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:07:04.749643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.744651 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:07:04.749643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.746289 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:07:04.749643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.748156 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:07:04.785452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.784909 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:07:04.786298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.786276 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:07:04.787113 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:04.787091 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:07:05.019672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:05.019560 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg"] Nov 03 21:07:05.022941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:05.022913 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-d78b90-hdgtg"] Nov 03 21:07:05.110728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:05.110697 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:07:05.114052 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:05.114028 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:07:05.117283 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:05.117258 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:07:05.904302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:05.904263 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd"] Nov 03 21:07:06.019825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:06.019776 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq"] Nov 03 21:07:06.052750 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:06.052716 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce949650-dc23-4e4d-9865-cd44ab218709" path="/var/lib/kubelet/pods/ce949650-dc23-4e4d-9865-cd44ab218709/volumes" Nov 03 21:07:10.906645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:10.904710 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" podUID="fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" containerName="payload-processing" containerID="cri-o://e6a93140c37ff36b18d99ebe8a9fff40b408939a7a225e6ff5c1a6fc765ce3d4" gracePeriod=25 Nov 03 21:07:11.020913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.020753 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" podUID="c63ff093-d891-4d35-8881-f24ae99fc54c" containerName="payload-pre-processing" containerID="cri-o://09c4691bfe610da30510188bd3996d32ffb7d4a24f34256aab195a3ee53a8d06" gracePeriod=25 Nov 03 21:07:11.309678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.295555 5932 generic.go:356] "Generic (PLEG): container finished" podID="c63ff093-d891-4d35-8881-f24ae99fc54c" containerID="09c4691bfe610da30510188bd3996d32ffb7d4a24f34256aab195a3ee53a8d06" exitCode=0 Nov 03 21:07:11.309678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.295835 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" event={"ID":"c63ff093-d891-4d35-8881-f24ae99fc54c","Type":"ContainerDied","Data":"09c4691bfe610da30510188bd3996d32ffb7d4a24f34256aab195a3ee53a8d06"} Nov 03 21:07:11.309678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.306675 5932 generic.go:356] "Generic (PLEG): container finished" podID="fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" containerID="e6a93140c37ff36b18d99ebe8a9fff40b408939a7a225e6ff5c1a6fc765ce3d4" exitCode=0 Nov 03 21:07:11.309678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.306798 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" event={"ID":"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b","Type":"ContainerDied","Data":"e6a93140c37ff36b18d99ebe8a9fff40b408939a7a225e6ff5c1a6fc765ce3d4"} Nov 03 21:07:12.028910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.856198 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:12.028910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.856704 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:12.028910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.905761 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:12.028910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.958757 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 21:07:12.028910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:11.993809 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:12.028910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:12.012948 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 21:07:12.608962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:12.239997 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 21:07:12.608962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:12.243709 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 03 21:07:12.772085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:12.749178 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:12.865878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:12.865754 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:12.920769 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:12.920678 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:13.199691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.128713 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 21:07:13.199691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.129085 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:07:13.199691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.129514 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": context deadline exceeded" Nov 03 21:07:13.319704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.316893 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:13.319704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.317353 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 21:07:13.435649 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:13.412493 5932 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.342s" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480135 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480229 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480293 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480354 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480376 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c"] Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480554 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480616 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.480691 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:07:13.546703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.536456 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" containerID="cri-o://cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b" gracePeriod=30 Nov 03 21:07:13.825711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.813222 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 21:07:13.983034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:13.982997 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 21:07:14.031457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.031416 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 21:07:14.098512 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.098421 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" Nov 03 21:07:14.128239 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.128196 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume\") pod \"c63ff093-d891-4d35-8881-f24ae99fc54c\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " Nov 03 21:07:14.128505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.128478 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d\" (UniqueName: \"kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d\") pod \"c63ff093-d891-4d35-8881-f24ae99fc54c\" (UID: \"c63ff093-d891-4d35-8881-f24ae99fc54c\") " Nov 03 21:07:14.128835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.128804 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9\" (UniqueName: \"kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9\") pod \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " Nov 03 21:07:14.129060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.129043 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume\") pod \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\" (UID: \"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b\") " Nov 03 21:07:14.162649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.144695 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:14.163348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.144734 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:07:14.207529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.205312 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d" pod "c63ff093-d891-4d35-8881-f24ae99fc54c" (UID: "c63ff093-d891-4d35-8881-f24ae99fc54c"). InnerVolumeSpecName "kube-api-access-pbt9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:14.207529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.194700 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:14.208552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.208521 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume" pod "fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" (UID: "fcd320fa-0802-4b5a-8c9f-32cd9f43b59b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:14.211294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.211261 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.109:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:14.214783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.214204 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume" pod "c63ff093-d891-4d35-8881-f24ae99fc54c" (UID: "c63ff093-d891-4d35-8881-f24ae99fc54c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:14.214783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.214482 5932 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:07:14.214783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.214570 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9" pod "fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" (UID: "fcd320fa-0802-4b5a-8c9f-32cd9f43b59b"). InnerVolumeSpecName "kube-api-access-gdxk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:14.262578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.262543 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c63ff093-d891-4d35-8881-f24ae99fc54c-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:14.262849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.262831 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbt9d\" (UniqueName: \"kubernetes.io/projected/c63ff093-d891-4d35-8881-f24ae99fc54c-kube-api-access-pbt9d\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:14.262959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.262946 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxk9\" (UniqueName: \"kubernetes.io/projected/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-kube-api-access-gdxk9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:14.263063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.263049 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:14.286008 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:14.285968 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:07:14.289755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.289711 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" Nov 03 21:07:14.291284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.291263 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:07:14.291534 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.291515 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq" event={"ID":"c63ff093-d891-4d35-8881-f24ae99fc54c","Type":"ContainerDied","Data":"a12f6fe6d11e469d53a04c2ae7212dba4f9ce3faa1cc00fce3c675c8fd2b73bc"} Nov 03 21:07:14.291688 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.291670 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd" event={"ID":"fcd320fa-0802-4b5a-8c9f-32cd9f43b59b","Type":"ContainerDied","Data":"db58b94152da7e583d89b04e1e68a28b2cebefa29285b44b9e1b80091c2dd575"} Nov 03 21:07:14.291817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.291804 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 21:07:14.295206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.295169 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 21:07:14.297934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.297888 5932 scope.go:122] "RemoveContainer" containerID="09c4691bfe610da30510188bd3996d32ffb7d4a24f34256aab195a3ee53a8d06" Nov 03 21:07:14.298246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.298107 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:07:14.298246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.298165 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:07:14.301197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.301181 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:07:14.303313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.303168 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 21:07:14.304939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.304923 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 21:07:14.318426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.318405 5932 scope.go:122] "RemoveContainer" containerID="e6a93140c37ff36b18d99ebe8a9fff40b408939a7a225e6ff5c1a6fc765ce3d4" Nov 03 21:07:14.676921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.676885 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn"] Nov 03 21:07:14.677119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.677065 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" podUID="33bce675-6c82-4160-b9c1-8f5a1ecdb419" containerName="maas-api" containerID="cri-o://c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92" gracePeriod=30 Nov 03 21:07:14.877224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.877178 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd"] Nov 03 21:07:14.886282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:14.886256 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-cc5d0f-695dccc48f-8vnbd"] Nov 03 21:07:15.092529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.092467 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:07:15.210643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.208809 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca\") pod \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " Nov 03 21:07:15.210643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.208942 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls\") pod \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " Nov 03 21:07:15.210643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.208978 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8\" (UniqueName: \"kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8\") pod \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " Nov 03 21:07:15.210643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.209112 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs\") pod \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\" (UID: \"33bce675-6c82-4160-b9c1-8f5a1ecdb419\") " Nov 03 21:07:15.210643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.209825 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca" pod "33bce675-6c82-4160-b9c1-8f5a1ecdb419" (UID: "33bce675-6c82-4160-b9c1-8f5a1ecdb419"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:15.211372 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.211118 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33bce675-6c82-4160-b9c1-8f5a1ecdb419-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:15.220734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.218485 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls" pod "33bce675-6c82-4160-b9c1-8f5a1ecdb419" (UID: "33bce675-6c82-4160-b9c1-8f5a1ecdb419"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:15.305694 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.305562 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs" pod "33bce675-6c82-4160-b9c1-8f5a1ecdb419" (UID: "33bce675-6c82-4160-b9c1-8f5a1ecdb419"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:07:15.308180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.307239 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8" pod "33bce675-6c82-4160-b9c1-8f5a1ecdb419" (UID: "33bce675-6c82-4160-b9c1-8f5a1ecdb419"). InnerVolumeSpecName "kube-api-access-7j5b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:15.313555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.312803 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:15.313555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.312835 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j5b8\" (UniqueName: \"kubernetes.io/projected/33bce675-6c82-4160-b9c1-8f5a1ecdb419-kube-api-access-7j5b8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:15.313555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.312850 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33bce675-6c82-4160-b9c1-8f5a1ecdb419-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:15.367536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.367494 5932 generic.go:356] "Generic (PLEG): container finished" podID="33bce675-6c82-4160-b9c1-8f5a1ecdb419" containerID="c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92" exitCode=0 Nov 03 21:07:15.367764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.367667 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" Nov 03 21:07:15.367764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.367693 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" event={"ID":"33bce675-6c82-4160-b9c1-8f5a1ecdb419","Type":"ContainerDied","Data":"c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92"} Nov 03 21:07:15.367764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.367742 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn" event={"ID":"33bce675-6c82-4160-b9c1-8f5a1ecdb419","Type":"ContainerDied","Data":"cab6f81935815673aa5d8317d3e057cbc068b989a5ccba903348248ce190f02f"} Nov 03 21:07:15.367944 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.367769 5932 scope.go:122] "RemoveContainer" containerID="c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92" Nov 03 21:07:15.389976 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.387929 5932 scope.go:122] "RemoveContainer" containerID="c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92" Nov 03 21:07:15.390355 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:15.390322 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92\": container with ID starting with c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92 not found: ID does not exist" containerID="c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92" Nov 03 21:07:15.390513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:15.390364 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92"} err="failed to get container status \"c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92\": rpc error: code = NotFound desc = could not find container \"c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92\": container with ID starting with c678d06e2dd6a8327496788b29d4b7bf64c3d9b75e5702b8aea30657564e7b92 not found: ID does not exist" Nov 03 21:07:16.006201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.006176 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq"] Nov 03 21:07:16.011152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.011129 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-cc5d0f-95f497f67-5xkrq"] Nov 03 21:07:16.067571 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.064662 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c63ff093-d891-4d35-8881-f24ae99fc54c" path="/var/lib/kubelet/pods/c63ff093-d891-4d35-8881-f24ae99fc54c/volumes" Nov 03 21:07:16.067571 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.065338 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" path="/var/lib/kubelet/pods/fcd320fa-0802-4b5a-8c9f-32cd9f43b59b/volumes" Nov 03 21:07:16.529071 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.529035 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn"] Nov 03 21:07:16.533479 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.533452 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8448c9-76d5469dcd-xdphn"] Nov 03 21:07:16.990806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.990770 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d"] Nov 03 21:07:16.994465 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:16.994439 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-8448c9-cp58d"] Nov 03 21:07:18.054981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:18.054949 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33bce675-6c82-4160-b9c1-8f5a1ecdb419" path="/var/lib/kubelet/pods/33bce675-6c82-4160-b9c1-8f5a1ecdb419/volumes" Nov 03 21:07:18.055513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:18.055496 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf87bd98-07c6-49fc-8880-faeb56ed372c" path="/var/lib/kubelet/pods/bf87bd98-07c6-49fc-8880-faeb56ed372c/volumes" Nov 03 21:07:19.051063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.051036 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:07:19.107090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107056 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107120 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107193 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107218 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107253 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107275 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107324 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107404 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.107529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107493 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket\") pod \"c9e11e9a-8524-4256-b778-5504f5167180\" (UID: \"c9e11e9a-8524-4256-b778-5504f5167180\") " Nov 03 21:07:19.108122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107585 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:19.108122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107604 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:07:19.108122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107797 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:19.108122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107878 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:19.108122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.107919 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:19.108415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.108127 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.108415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.108149 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.108415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.108164 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.108415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.108179 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.108415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.108194 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e11e9a-8524-4256-b778-5504f5167180-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.109614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.109586 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:07:19.109614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.109645 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:07:19.109814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.109789 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "kube-api-access-hgrbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:19.109859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.109826 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token" pod "c9e11e9a-8524-4256-b778-5504f5167180" (UID: "c9e11e9a-8524-4256-b778-5504f5167180"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:07:19.209607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.209563 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.209607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.209607 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e11e9a-8524-4256-b778-5504f5167180-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.210070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.209639 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgrbw\" (UniqueName: \"kubernetes.io/projected/c9e11e9a-8524-4256-b778-5504f5167180-kube-api-access-hgrbw\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.210070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.209657 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e11e9a-8524-4256-b778-5504f5167180-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:07:19.410224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.410183 5932 generic.go:356] "Generic (PLEG): container finished" podID="c9e11e9a-8524-4256-b778-5504f5167180" containerID="cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b" exitCode=0 Nov 03 21:07:19.410224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.410229 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" Nov 03 21:07:19.410421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.410274 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" event={"ID":"c9e11e9a-8524-4256-b778-5504f5167180","Type":"ContainerDied","Data":"cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b"} Nov 03 21:07:19.410421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.410360 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c" event={"ID":"c9e11e9a-8524-4256-b778-5504f5167180","Type":"ContainerDied","Data":"0e9f7cb54c33ee4c90a9894c383e340046a9e261525b1a8f830e853a553ef517"} Nov 03 21:07:19.410421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.410387 5932 scope.go:122] "RemoveContainer" containerID="cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b" Nov 03 21:07:19.422769 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.422745 5932 scope.go:122] "RemoveContainer" containerID="0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323" Nov 03 21:07:19.433507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.433481 5932 scope.go:122] "RemoveContainer" containerID="cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b" Nov 03 21:07:19.433843 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:19.433820 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b\": container with ID starting with cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b not found: ID does not exist" containerID="cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b" Nov 03 21:07:19.433919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.433851 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b"} err="failed to get container status \"cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b\": rpc error: code = NotFound desc = could not find container \"cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b\": container with ID starting with cb073baf9e28c4dc34e084681bb1f1e698954a61053a0f15e9c3994d612c2e1b not found: ID does not exist" Nov 03 21:07:19.433919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.433872 5932 scope.go:122] "RemoveContainer" containerID="0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323" Nov 03 21:07:19.434160 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:19.434144 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323\": container with ID starting with 0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323 not found: ID does not exist" containerID="0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323" Nov 03 21:07:19.434201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.434165 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323"} err="failed to get container status \"0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323\": rpc error: code = NotFound desc = could not find container \"0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323\": container with ID starting with 0fa3e3b1ba8fbf45cb3073f6191851510c0d43ef68b58319fc4b63a4395c7323 not found: ID does not exist" Nov 03 21:07:19.566574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.566534 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c"] Nov 03 21:07:19.572832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:19.572806 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d78b90-openshift-default-7fc4b57745-qx94c"] Nov 03 21:07:20.053398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:20.053365 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e11e9a-8524-4256-b778-5504f5167180" path="/var/lib/kubelet/pods/c9e11e9a-8524-4256-b778-5504f5167180/volumes" Nov 03 21:07:23.791063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.791024 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf"] Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792151 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792236 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792273 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c63ff093-d891-4d35-8881-f24ae99fc54c" containerName="payload-pre-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792282 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c63ff093-d891-4d35-8881-f24ae99fc54c" containerName="payload-pre-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792298 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792306 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792323 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" containerName="payload-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792334 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" containerName="payload-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792354 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6a22bcae-cd8d-4522-bf24-0392d534bfc9" containerName="maas-api" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792360 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a22bcae-cd8d-4522-bf24-0392d534bfc9" containerName="maas-api" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792374 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2c053419-7452-4329-8ff0-71eff6aa8c02" containerName="payload-pre-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792383 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c053419-7452-4329-8ff0-71eff6aa8c02" containerName="payload-pre-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792396 5932 cpu_manager.go:414] "containerMap: removing container" podUID="33bce675-6c82-4160-b9c1-8f5a1ecdb419" containerName="maas-api" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792404 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33bce675-6c82-4160-b9c1-8f5a1ecdb419" containerName="maas-api" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792450 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f2d1bece-24b9-4202-a5c6-285f3e99834a" containerName="payload-processing" Nov 03 21:07:23.792577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792458 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d1bece-24b9-4202-a5c6-285f3e99834a" containerName="payload-processing" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792645 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd320fa-0802-4b5a-8c9f-32cd9f43b59b" containerName="payload-processing" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792663 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a22bcae-cd8d-4522-bf24-0392d534bfc9" containerName="maas-api" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792675 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c053419-7452-4329-8ff0-71eff6aa8c02" containerName="payload-pre-processing" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792685 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="33bce675-6c82-4160-b9c1-8f5a1ecdb419" containerName="maas-api" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792702 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d1bece-24b9-4202-a5c6-285f3e99834a" containerName="payload-processing" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792719 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" Nov 03 21:07:23.793500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.792731 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c63ff093-d891-4d35-8881-f24ae99fc54c" containerName="payload-pre-processing" Nov 03 21:07:23.799040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.799012 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.860203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.860163 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7a3593-openshift-default-dockercfg-qdw9k\"" Nov 03 21:07:23.876997 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.876964 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877081 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877176 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877256 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877360 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877345 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877403 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877370 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877405 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877477 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gcm\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:23.877529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:23.877521 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.003646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.003540 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.003646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.003597 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.054848 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.054917 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.054994 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.055049 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gcm\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.055114 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.055220 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.055630 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.055302 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.068969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.067162 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf"] Nov 03 21:07:24.069149 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.069128 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.069716 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.069690 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.071390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.071359 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.071979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.071951 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.072234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.072210 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.072438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.072415 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.073256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.073235 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.132320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.132278 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.138350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.138315 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gcm\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm\") pod \"e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.417337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.417302 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:24.854732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.854582 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf"] Nov 03 21:07:24.859583 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:24.859555 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486253bb_d0a1_4d28_a087_c132a05238ea.slice/crio-fcf2cfea89e547cbc71ac45d90bbd92aad25c09e7dd46c317cf41b3abad4b461 WatchSource:0}: Error finding container fcf2cfea89e547cbc71ac45d90bbd92aad25c09e7dd46c317cf41b3abad4b461: Status 404 returned error can't find the container with id fcf2cfea89e547cbc71ac45d90bbd92aad25c09e7dd46c317cf41b3abad4b461 Nov 03 21:07:24.862291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.862255 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:24.862379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.862328 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:24.862379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:24.862359 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:25.463342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:25.463301 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" event={"ID":"486253bb-d0a1-4d28-a087-c132a05238ea","Type":"ContainerStarted","Data":"b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe"} Nov 03 21:07:25.463526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:25.463369 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" event={"ID":"486253bb-d0a1-4d28-a087-c132a05238ea","Type":"ContainerStarted","Data":"fcf2cfea89e547cbc71ac45d90bbd92aad25c09e7dd46c317cf41b3abad4b461"} Nov 03 21:07:25.577896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:25.577842 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podStartSLOduration=2.577828262 podStartE2EDuration="2.577828262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:25.575604953 +0000 UTC m=+2253.640846428" watchObservedRunningTime="2026-11-03 21:07:25.577828262 +0000 UTC m=+2253.643069742" Nov 03 21:07:26.418319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:26.418280 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:26.425822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:26.425791 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:26.471750 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:26.471710 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:26.473056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:26.473029 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:07:32.099256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.099216 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g"] Nov 03 21:07:32.100027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.100010 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e11e9a-8524-4256-b778-5504f5167180" containerName="istio-proxy" Nov 03 21:07:32.104865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.104842 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.157287 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.157252 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g"] Nov 03 21:07:32.202121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.199593 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-7a3593-dockercfg-dxfrm\"" Nov 03 21:07:32.212155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.208725 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-7a3593\"" Nov 03 21:07:32.366369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.299687 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f"] Nov 03 21:07:32.366369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.363694 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hthr\" (UniqueName: \"kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr\") pod \"payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.366369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.363760 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.366369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.365760 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.389821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.389742 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f"] Nov 03 21:07:32.465090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.465047 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6klbd\" (UniqueName: \"kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd\") pod \"payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.465297 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.465224 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.465373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.465350 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hthr\" (UniqueName: \"kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr\") pod \"payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.465444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.465391 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.466163 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.466143 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.525197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.525162 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hthr\" (UniqueName: \"kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr\") pod \"payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:32.566428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.566393 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6klbd\" (UniqueName: \"kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd\") pod \"payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.566664 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.566510 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.567348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.567326 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.625805 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.625776 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6klbd\" (UniqueName: \"kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd\") pod \"payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.693980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.693898 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:32.720554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:32.720528 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:33.101414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.101327 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f"] Nov 03 21:07:33.106257 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:33.106227 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eaa62d1_42a9_414f_8370_276e039fde3b.slice/crio-389b263660edcf7cae93055c47fb179dea11b2701dd576c76b782a7ca0a59fc1 WatchSource:0}: Error finding container 389b263660edcf7cae93055c47fb179dea11b2701dd576c76b782a7ca0a59fc1: Status 404 returned error can't find the container with id 389b263660edcf7cae93055c47fb179dea11b2701dd576c76b782a7ca0a59fc1 Nov 03 21:07:33.213296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.213258 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g"] Nov 03 21:07:33.224010 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:33.223972 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ac0eef_b9ac_4df3_a6e7_abb72fe741e3.slice/crio-2ce77033ebf529f53ed01969c7d05985503b0c07bc51a839dc61268dee5c337b WatchSource:0}: Error finding container 2ce77033ebf529f53ed01969c7d05985503b0c07bc51a839dc61268dee5c337b: Status 404 returned error can't find the container with id 2ce77033ebf529f53ed01969c7d05985503b0c07bc51a839dc61268dee5c337b Nov 03 21:07:33.530442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.530399 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" event={"ID":"5eaa62d1-42a9-414f-8370-276e039fde3b","Type":"ContainerStarted","Data":"dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4"} Nov 03 21:07:33.530706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.530468 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" event={"ID":"5eaa62d1-42a9-414f-8370-276e039fde3b","Type":"ContainerStarted","Data":"389b263660edcf7cae93055c47fb179dea11b2701dd576c76b782a7ca0a59fc1"} Nov 03 21:07:33.530706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.530495 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:33.532354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.532320 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" event={"ID":"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3","Type":"ContainerStarted","Data":"45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd"} Nov 03 21:07:33.532485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.532374 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" event={"ID":"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3","Type":"ContainerStarted","Data":"2ce77033ebf529f53ed01969c7d05985503b0c07bc51a839dc61268dee5c337b"} Nov 03 21:07:33.532485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.532427 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:33.644065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.644011 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" podStartSLOduration=1.6439902480000002 podStartE2EDuration="1.643990248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:33.639874646 +0000 UTC m=+2261.705116153" watchObservedRunningTime="2026-11-03 21:07:33.643990248 +0000 UTC m=+2261.709231720" Nov 03 21:07:33.769645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:33.755112 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" podStartSLOduration=1.755092307 podStartE2EDuration="1.755092307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:33.74768565 +0000 UTC m=+2261.812927127" watchObservedRunningTime="2026-11-03 21:07:33.755092307 +0000 UTC m=+2261.820333781" Nov 03 21:07:35.938525 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:35.938487 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq"] Nov 03 21:07:35.943146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:35.943122 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.001706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.000983 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8afb98-openshift-default-dockercfg-j9pwk\"" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.059739 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9cgj\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.059826 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.059911 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.059959 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.060036 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.060096 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.060190 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.060248 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.065652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.060305 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.165959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.165919 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.166302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.166275 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.173671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.173356 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq"] Nov 03 21:07:36.175714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.174888 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.175966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.175947 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.176113 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.176099 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.181645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.176318 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9cgj\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.181937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.181908 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.182142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.182122 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.182298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.182278 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.182948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.182919 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.229743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.196100 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.229743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.197074 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.229743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.197736 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.229743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.208107 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.257122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.257052 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.258172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.258118 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9cgj\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.262650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.262083 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.262817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.262652 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo\") pod \"e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.298243 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.298215 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:36.738498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.738467 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq"] Nov 03 21:07:36.741123 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:36.741093 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e6377a_7d4b_4dde_9d95_9c3b271a095b.slice/crio-aff68b152d7dd00f0cc25e1deb3a5fe7c89e0e9f654bbb7f123a5d2ab8d10a9b WatchSource:0}: Error finding container aff68b152d7dd00f0cc25e1deb3a5fe7c89e0e9f654bbb7f123a5d2ab8d10a9b: Status 404 returned error can't find the container with id aff68b152d7dd00f0cc25e1deb3a5fe7c89e0e9f654bbb7f123a5d2ab8d10a9b Nov 03 21:07:36.744011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.743976 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:36.744093 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.744049 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:36.744148 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:36.744089 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:37.570937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:37.570899 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" event={"ID":"d0e6377a-7d4b-4dde-9d95-9c3b271a095b","Type":"ContainerStarted","Data":"46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee"} Nov 03 21:07:37.571340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:37.570957 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" event={"ID":"d0e6377a-7d4b-4dde-9d95-9c3b271a095b","Type":"ContainerStarted","Data":"aff68b152d7dd00f0cc25e1deb3a5fe7c89e0e9f654bbb7f123a5d2ab8d10a9b"} Nov 03 21:07:37.689868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:37.689804 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podStartSLOduration=2.6897863600000003 podStartE2EDuration="2.68978636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:37.686303749 +0000 UTC m=+2265.751545221" watchObservedRunningTime="2026-11-03 21:07:37.68978636 +0000 UTC m=+2265.755027835" Nov 03 21:07:38.298847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:38.298811 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:38.305570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:38.305540 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:38.578446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:38.578369 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:38.579546 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:38.579523 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:07:40.703971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.703933 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj"] Nov 03 21:07:40.708809 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.708781 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.808110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.808064 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj"] Nov 03 21:07:40.871810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.871773 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.871810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.871815 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.871835 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.871932 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgt84\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.872011 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.872090 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.872162 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872254 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.872238 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.872291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.872278 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973242 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973290 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973310 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973353 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgt84\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973382 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973456 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973558 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.973703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973657 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.974065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973715 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.974065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973785 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.974065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973898 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.974065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.973967 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.974065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.974060 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.974413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.974389 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.975803 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.975784 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:40.975947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:40.975928 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:41.031129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.031092 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:41.031318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.031188 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgt84\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:41.325573 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.325481 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:41.761508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.761456 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj"] Nov 03 21:07:41.802102 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:41.802071 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda10562d1_1774_4086_90ee_71cf548fd839.slice/crio-8f1cb96eb4527613ee01bd6e8d682996cb5e361f1f87b2bf15f9f83095f5a454 WatchSource:0}: Error finding container 8f1cb96eb4527613ee01bd6e8d682996cb5e361f1f87b2bf15f9f83095f5a454: Status 404 returned error can't find the container with id 8f1cb96eb4527613ee01bd6e8d682996cb5e361f1f87b2bf15f9f83095f5a454 Nov 03 21:07:41.804740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.804702 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:41.804870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.804790 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:41.804870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:41.804837 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:42.612001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:42.611962 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" event={"ID":"a10562d1-1774-4086-90ee-71cf548fd839","Type":"ContainerStarted","Data":"dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71"} Nov 03 21:07:42.612192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:42.612020 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" event={"ID":"a10562d1-1774-4086-90ee-71cf548fd839","Type":"ContainerStarted","Data":"8f1cb96eb4527613ee01bd6e8d682996cb5e361f1f87b2bf15f9f83095f5a454"} Nov 03 21:07:42.775171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:42.775111 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podStartSLOduration=2.775094267 podStartE2EDuration="2.775094267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:42.771882412 +0000 UTC m=+2270.837123885" watchObservedRunningTime="2026-11-03 21:07:42.775094267 +0000 UTC m=+2270.840335740" Nov 03 21:07:43.345648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:43.342024 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:43.345869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:43.345828 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 21:07:43.345950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:43.345902 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 21:07:44.178814 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:07:44.178768 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:07:44.343657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:44.327023 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 21:07:44.343657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:44.327087 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 21:07:44.550154 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:44.550036 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:07:44.550723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:44.550697 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:07:45.342648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:45.335722 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 21:07:45.342648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:45.335802 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 21:07:46.343180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:46.343135 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 21:07:46.343817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:46.343780 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 21:07:47.345790 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:47.345738 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 21:07:47.346357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:47.345845 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 21:07:48.328848 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:48.328809 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 21:07:48.329784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:48.329749 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 21:07:49.332817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:49.332779 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:49.333602 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:49.333575 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:49.334888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:49.334869 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:07:55.827666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.827617 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs"] Nov 03 21:07:55.833452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.833426 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.927234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.927197 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs"] Nov 03 21:07:55.938674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.938601 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.938989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.938957 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939082 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vndxd\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939213 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939408 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939384 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939480 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939433 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939538 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939644 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939608 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:55.939742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:55.939723 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.040645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040592 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.040850 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040699 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vndxd\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.040850 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040745 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.040850 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040843 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.041044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040881 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.041044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040931 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.041044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.040980 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.041228 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.041063 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.041228 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.041112 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.069548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.069507 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.069773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.069586 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.070192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.069868 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.070192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.070127 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.077914 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.077837 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.083834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.083798 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.084540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.084515 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.099276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.099222 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vndxd\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.118282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.118245 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token\") pod \"e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.147467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.147426 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:56.583811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.583775 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs"] Nov 03 21:07:56.588646 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:07:56.588599 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41f1a494_1627_4a59_9e2d_2d54db5545c1.slice/crio-f13b828cd7475959c2792a04f16152bb720edaa47c8523f1dd51244c3f06e937 WatchSource:0}: Error finding container f13b828cd7475959c2792a04f16152bb720edaa47c8523f1dd51244c3f06e937: Status 404 returned error can't find the container with id f13b828cd7475959c2792a04f16152bb720edaa47c8523f1dd51244c3f06e937 Nov 03 21:07:56.591547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.591512 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:56.591677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.591585 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:56.591677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.591662 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:07:56.801550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.801501 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" event={"ID":"41f1a494-1627-4a59-9e2d-2d54db5545c1","Type":"ContainerStarted","Data":"96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a"} Nov 03 21:07:56.801761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.801566 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" event={"ID":"41f1a494-1627-4a59-9e2d-2d54db5545c1","Type":"ContainerStarted","Data":"f13b828cd7475959c2792a04f16152bb720edaa47c8523f1dd51244c3f06e937"} Nov 03 21:07:56.918888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:56.918828 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podStartSLOduration=1.918811464 podStartE2EDuration="1.918811464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:07:56.914214627 +0000 UTC m=+2284.979456124" watchObservedRunningTime="2026-11-03 21:07:56.918811464 +0000 UTC m=+2284.984052937" Nov 03 21:07:57.148329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:57.148280 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:07:57.149893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:57.149862 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:07:57.150025 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:57.149923 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:07:58.148252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:58.148216 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:07:58.148702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:58.148273 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:07:59.148313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:59.148268 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:07:59.148885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:59.148345 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:07:59.407179 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:59.407083 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g"] Nov 03 21:07:59.407364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:07:59.407288 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" podUID="49438490-df7e-4267-90af-57c0b9394625" containerName="maas-api" containerID="cri-o://f82979e9e3b13cdb3f2ac9e69a99d9edaf26332f3ea82ec53eef1ca8a1a46478" gracePeriod=30 Nov 03 21:08:00.003743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.001050 5932 generic.go:356] "Generic (PLEG): container finished" podID="49438490-df7e-4267-90af-57c0b9394625" containerID="f82979e9e3b13cdb3f2ac9e69a99d9edaf26332f3ea82ec53eef1ca8a1a46478" exitCode=0 Nov 03 21:08:00.003743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.001130 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" event={"ID":"49438490-df7e-4267-90af-57c0b9394625","Type":"ContainerDied","Data":"f82979e9e3b13cdb3f2ac9e69a99d9edaf26332f3ea82ec53eef1ca8a1a46478"} Nov 03 21:08:00.071658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.071602 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:08:00.148322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.148285 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:08:00.148536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.148339 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:08:00.156901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.156859 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\") pod \"49438490-df7e-4267-90af-57c0b9394625\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " Nov 03 21:08:00.157053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.156931 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs\") pod \"49438490-df7e-4267-90af-57c0b9394625\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " Nov 03 21:08:00.157053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.156972 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc\" (UniqueName: \"kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc\") pod \"49438490-df7e-4267-90af-57c0b9394625\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " Nov 03 21:08:00.157169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.157160 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca\") pod \"49438490-df7e-4267-90af-57c0b9394625\" (UID: \"49438490-df7e-4267-90af-57c0b9394625\") " Nov 03 21:08:00.157716 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.157617 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca" pod "49438490-df7e-4267-90af-57c0b9394625" (UID: "49438490-df7e-4267-90af-57c0b9394625"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:00.157849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.157828 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49438490-df7e-4267-90af-57c0b9394625-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:00.159228 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.159197 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs" pod "49438490-df7e-4267-90af-57c0b9394625" (UID: "49438490-df7e-4267-90af-57c0b9394625"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:00.159228 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.159215 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc" pod "49438490-df7e-4267-90af-57c0b9394625" (UID: "49438490-df7e-4267-90af-57c0b9394625"). InnerVolumeSpecName "kube-api-access-hj2vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:00.159430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.159415 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls" pod "49438490-df7e-4267-90af-57c0b9394625" (UID: "49438490-df7e-4267-90af-57c0b9394625"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:00.263607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.263493 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:00.263607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.263544 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49438490-df7e-4267-90af-57c0b9394625-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:00.263607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:00.263555 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj2vc\" (UniqueName: \"kubernetes.io/projected/49438490-df7e-4267-90af-57c0b9394625-kube-api-access-hj2vc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:01.010489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.010455 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" event={"ID":"49438490-df7e-4267-90af-57c0b9394625","Type":"ContainerDied","Data":"d1813e66769d64af669f03bb0d8cabf88243c8f8cb2f09535683b9fc6b8413a3"} Nov 03 21:08:01.010489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.010483 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g" Nov 03 21:08:01.010783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.010518 5932 scope.go:122] "RemoveContainer" containerID="f82979e9e3b13cdb3f2ac9e69a99d9edaf26332f3ea82ec53eef1ca8a1a46478" Nov 03 21:08:01.151179 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.151144 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:08:01.151763 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.151206 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:08:01.170700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.170657 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g"] Nov 03 21:08:01.175952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.175923 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw"] Nov 03 21:08:01.176669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.176654 5932 cpu_manager.go:414] "containerMap: removing container" podUID="49438490-df7e-4267-90af-57c0b9394625" containerName="maas-api" Nov 03 21:08:01.176752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.176671 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49438490-df7e-4267-90af-57c0b9394625" containerName="maas-api" Nov 03 21:08:01.176795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.176789 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="49438490-df7e-4267-90af-57c0b9394625" containerName="maas-api" Nov 03 21:08:01.181902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.181881 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.214861 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.214826 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0776c6-67c664544c-bgz4g"] Nov 03 21:08:01.230826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.230658 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:01.230826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.230793 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:08:01.281290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.281200 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.293778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.289005 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmp5f\" (UniqueName: \"kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.293778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.289214 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.293778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.291324 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw"] Nov 03 21:08:01.410656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.407260 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.410656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.407453 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.410656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.407533 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmp5f\" (UniqueName: \"kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.498649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.497960 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.570967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.570762 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmp5f\" (UniqueName: \"kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.576985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.572181 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d150fa62-gs6nw\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:01.797952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:01.797885 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:02.066158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.066119 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49438490-df7e-4267-90af-57c0b9394625" path="/var/lib/kubelet/pods/49438490-df7e-4267-90af-57c0b9394625/volumes" Nov 03 21:08:02.148417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.148322 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:08:02.148417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.148382 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:08:02.300432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.300393 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw"] Nov 03 21:08:02.305913 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:02.305881 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dd41dc_505a_4b99_83e5_f2f9be4db6aa.slice/crio-58b78e56f38a866c6504b858690872f83c68510d0e9d5cc509be023303b33232 WatchSource:0}: Error finding container 58b78e56f38a866c6504b858690872f83c68510d0e9d5cc509be023303b33232: Status 404 returned error can't find the container with id 58b78e56f38a866c6504b858690872f83c68510d0e9d5cc509be023303b33232 Nov 03 21:08:02.582456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.582360 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67"] Nov 03 21:08:02.590832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.590792 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:02.598938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.598909 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67"] Nov 03 21:08:02.649651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.645924 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8afb98\"" Nov 03 21:08:02.660566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.653876 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8afb98-dockercfg-nngh8\"" Nov 03 21:08:02.741649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.730200 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw"] Nov 03 21:08:02.871774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.871675 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljd5\" (UniqueName: \"kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5\") pod \"payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:02.903669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.872215 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:02.921642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.912983 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:02.921642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:02.915062 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw"] Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.020006 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w"] Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.026187 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljd5\" (UniqueName: \"kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5\") pod \"payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.026432 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.026495 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.026561 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g67p2\" (UniqueName: \"kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2\") pod \"payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.026710 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.036668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.027927 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:03.052838 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.048944 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w"] Nov 03 21:08:03.060931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.059994 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" event={"ID":"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa","Type":"ContainerStarted","Data":"29068bbfc9f5e6d128aff50f0c99e21446cd9f6bea9ff12d578ccb3bbbbc586e"} Nov 03 21:08:03.060931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.060059 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" event={"ID":"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa","Type":"ContainerStarted","Data":"58b78e56f38a866c6504b858690872f83c68510d0e9d5cc509be023303b33232"} Nov 03 21:08:03.075961 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.075930 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fc4e8d-openshift-default-dockercfg-5zgwd\"" Nov 03 21:08:03.085117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.085081 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljd5\" (UniqueName: \"kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5\") pod \"payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:03.128398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128360 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.128638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128422 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.128638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128490 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.128804 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128669 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.128804 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128714 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfpb\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.128804 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128777 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g67p2\" (UniqueName: \"kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2\") pod \"payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.128988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128863 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.128988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.128909 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.129103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.129035 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.129103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.129069 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.129229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.129150 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.129326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.129306 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.148279 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.148236 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Nov 03 21:08:03.148469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.148286 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Nov 03 21:08:03.188325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.188284 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g67p2\" (UniqueName: \"kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2\") pod \"payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.209114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.209058 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257122 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257171 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257232 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257357 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257410 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257516 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257567 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfpb\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257635 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.257941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.257670 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.258571 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.258252 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.276648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.275741 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.276857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.276688 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.279041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.277207 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.282703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.279747 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.283917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.283677 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" podStartSLOduration=2.283659234 podStartE2EDuration="2.283659234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:03.268155123 +0000 UTC m=+2291.333396599" watchObservedRunningTime="2026-11-03 21:08:03.283659234 +0000 UTC m=+2291.348900708" Nov 03 21:08:03.286985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.286936 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.295528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.295475 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.316022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.315984 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.341850 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.341822 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfpb\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb\") pod \"e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.349396 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.349295 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:03.377051 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.377020 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:03.662844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.662808 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67"] Nov 03 21:08:03.703756 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:03.703701 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19d6694_9e7a_421e_9280_1836fee697c1.slice/crio-9d88eabad08edb7577349b40e6fb3808a780821fdd7483dec40dadcde654137f WatchSource:0}: Error finding container 9d88eabad08edb7577349b40e6fb3808a780821fdd7483dec40dadcde654137f: Status 404 returned error can't find the container with id 9d88eabad08edb7577349b40e6fb3808a780821fdd7483dec40dadcde654137f Nov 03 21:08:03.809439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.809399 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw"] Nov 03 21:08:03.813893 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:03.813863 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf318cf3f_313b_41f7_a5e4_6465fdd036ed.slice/crio-e7f945450939a33578647c9817463b57a069946e50d1622e2283b17464805ddd WatchSource:0}: Error finding container e7f945450939a33578647c9817463b57a069946e50d1622e2283b17464805ddd: Status 404 returned error can't find the container with id e7f945450939a33578647c9817463b57a069946e50d1622e2283b17464805ddd Nov 03 21:08:03.864692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.864650 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w"] Nov 03 21:08:03.867208 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:03.867178 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ab015e_f0c9_4936_8734_d1964c9cd4f1.slice/crio-69c02292bd11865f08f97f81198f5ef7f47c3ae46497239f3add4f1cd8f6ef1d WatchSource:0}: Error finding container 69c02292bd11865f08f97f81198f5ef7f47c3ae46497239f3add4f1cd8f6ef1d: Status 404 returned error can't find the container with id 69c02292bd11865f08f97f81198f5ef7f47c3ae46497239f3add4f1cd8f6ef1d Nov 03 21:08:03.870224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.870191 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:03.870439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.870263 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:03.870439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:03.870290 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:08:04.070444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.070402 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" event={"ID":"f318cf3f-313b-41f7-a5e4-6465fdd036ed","Type":"ContainerStarted","Data":"32a9effd3569c3b76f637ea5da16cbb26a26d632835e2fa7ebd5524c5f217ff6"} Nov 03 21:08:04.070662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.070469 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" event={"ID":"f318cf3f-313b-41f7-a5e4-6465fdd036ed","Type":"ContainerStarted","Data":"e7f945450939a33578647c9817463b57a069946e50d1622e2283b17464805ddd"} Nov 03 21:08:04.070662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.070501 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:04.072245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.072218 5932 generic.go:356] "Generic (PLEG): container finished" podID="f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" containerID="29068bbfc9f5e6d128aff50f0c99e21446cd9f6bea9ff12d578ccb3bbbbc586e" exitCode=0 Nov 03 21:08:04.072393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.072308 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" event={"ID":"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa","Type":"ContainerDied","Data":"29068bbfc9f5e6d128aff50f0c99e21446cd9f6bea9ff12d578ccb3bbbbc586e"} Nov 03 21:08:04.074182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.074154 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" event={"ID":"d7ab015e-f0c9-4936-8734-d1964c9cd4f1","Type":"ContainerStarted","Data":"5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401"} Nov 03 21:08:04.074302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.074202 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" event={"ID":"d7ab015e-f0c9-4936-8734-d1964c9cd4f1","Type":"ContainerStarted","Data":"69c02292bd11865f08f97f81198f5ef7f47c3ae46497239f3add4f1cd8f6ef1d"} Nov 03 21:08:04.075912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.075887 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" event={"ID":"f19d6694-9e7a-421e-9280-1836fee697c1","Type":"ContainerStarted","Data":"8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c"} Nov 03 21:08:04.076000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.075932 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" event={"ID":"f19d6694-9e7a-421e-9280-1836fee697c1","Type":"ContainerStarted","Data":"9d88eabad08edb7577349b40e6fb3808a780821fdd7483dec40dadcde654137f"} Nov 03 21:08:04.076070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.076015 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:04.154812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.154729 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:08:04.155207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.155131 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:08:04.156059 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.156035 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:08:04.179588 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.179531 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" podStartSLOduration=2.179516768 podStartE2EDuration="2.179516768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:04.178736117 +0000 UTC m=+2292.243977590" watchObservedRunningTime="2026-11-03 21:08:04.179516768 +0000 UTC m=+2292.244758240" Nov 03 21:08:04.377795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.377753 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:04.379518 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.379486 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:08:04.379728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.379538 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:08:04.412470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.412361 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" podStartSLOduration=2.412344097 podStartE2EDuration="2.412344097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:04.291105415 +0000 UTC m=+2292.356346906" watchObservedRunningTime="2026-11-03 21:08:04.412344097 +0000 UTC m=+2292.477585571" Nov 03 21:08:04.447664 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:04.446878 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podStartSLOduration=2.446854468 podStartE2EDuration="2.446854468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:04.410067459 +0000 UTC m=+2292.475308957" watchObservedRunningTime="2026-11-03 21:08:04.446854468 +0000 UTC m=+2292.512095938" Nov 03 21:08:05.272391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.272363 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:05.337789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.337692 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca\") pod \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " Nov 03 21:08:05.337965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.337847 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f\" (UniqueName: \"kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f\") pod \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " Nov 03 21:08:05.338038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.337969 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca\") pod \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\" (UID: \"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa\") " Nov 03 21:08:05.338122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.338090 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca" pod "f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" (UID: "f1dd41dc-505a-4b99-83e5-f2f9be4db6aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:05.338523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.338501 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:05.340193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.340162 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca" pod "f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" (UID: "f1dd41dc-505a-4b99-83e5-f2f9be4db6aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:05.340387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.340368 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f" pod "f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" (UID: "f1dd41dc-505a-4b99-83e5-f2f9be4db6aa"). InnerVolumeSpecName "kube-api-access-bmp5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:05.383489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.383452 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:05.439489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.439453 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmp5f\" (UniqueName: \"kubernetes.io/projected/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-kube-api-access-bmp5f\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:05.439489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:05.439489 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:06.098981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:06.098936 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" event={"ID":"f1dd41dc-505a-4b99-83e5-f2f9be4db6aa","Type":"ContainerDied","Data":"58b78e56f38a866c6504b858690872f83c68510d0e9d5cc509be023303b33232"} Nov 03 21:08:06.098981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:06.098972 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw" Nov 03 21:08:06.099238 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:06.099003 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b78e56f38a866c6504b858690872f83c68510d0e9d5cc509be023303b33232" Nov 03 21:08:06.099352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:06.099330 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:06.100351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:06.100331 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:08:08.052687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.044458 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487"] Nov 03 21:08:08.052687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.045686 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" containerName="revoke-keys" Nov 03 21:08:08.052687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.045713 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" containerName="revoke-keys" Nov 03 21:08:08.052687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.045953 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" containerName="revoke-keys" Nov 03 21:08:08.079645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.061848 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.099665 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.099599 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487"] Nov 03 21:08:08.107919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.107856 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.109945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.109868 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4bdz\" (UniqueName: \"kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.110100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.110069 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.150032 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.150006 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:08:08.150245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.150230 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:08.211952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.211915 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.212141 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.212027 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4bdz\" (UniqueName: \"kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.212194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.212148 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.212825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.212796 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.214538 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.214511 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.270424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.270384 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4bdz\" (UniqueName: \"kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz\") pod \"maas-api-revoke-keys-e2e-mt-e68f8112-p8487\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.401415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.401376 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:08.902132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:08.902092 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487"] Nov 03 21:08:08.907240 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:08.907168 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2a359d_5ea3_4193_8c83_f56491d63297.slice/crio-d715ecb09f7c28fa56a22a602b006af55b70d51ed3a291d561f714bd314880b8 WatchSource:0}: Error finding container d715ecb09f7c28fa56a22a602b006af55b70d51ed3a291d561f714bd314880b8: Status 404 returned error can't find the container with id d715ecb09f7c28fa56a22a602b006af55b70d51ed3a291d561f714bd314880b8 Nov 03 21:08:09.155882 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:09.155777 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" event={"ID":"9b2a359d-5ea3-4193-8c83-f56491d63297","Type":"ContainerStarted","Data":"d715ecb09f7c28fa56a22a602b006af55b70d51ed3a291d561f714bd314880b8"} Nov 03 21:08:10.168845 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:10.168755 5932 generic.go:356] "Generic (PLEG): container finished" podID="9b2a359d-5ea3-4193-8c83-f56491d63297" containerID="00418ae2a9e335e08321dd3cc9e94645ae34b7a9f96eba93749ca694dea71da0" exitCode=0 Nov 03 21:08:10.169227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:10.168896 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" event={"ID":"9b2a359d-5ea3-4193-8c83-f56491d63297","Type":"ContainerDied","Data":"00418ae2a9e335e08321dd3cc9e94645ae34b7a9f96eba93749ca694dea71da0"} Nov 03 21:08:10.170836 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:10.170805 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d"] Nov 03 21:08:10.171094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:10.171066 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" containerID="cri-o://f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b" gracePeriod=30 Nov 03 21:08:11.403645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.393221 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:11.508849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.508755 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca\") pod \"9b2a359d-5ea3-4193-8c83-f56491d63297\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " Nov 03 21:08:11.509951 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.509923 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz\" (UniqueName: \"kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz\") pod \"9b2a359d-5ea3-4193-8c83-f56491d63297\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " Nov 03 21:08:11.510608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.510579 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca\") pod \"9b2a359d-5ea3-4193-8c83-f56491d63297\" (UID: \"9b2a359d-5ea3-4193-8c83-f56491d63297\") " Nov 03 21:08:11.511219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.511102 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca" pod "9b2a359d-5ea3-4193-8c83-f56491d63297" (UID: "9b2a359d-5ea3-4193-8c83-f56491d63297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:11.512018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.511994 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:11.535473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.535406 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca" pod "9b2a359d-5ea3-4193-8c83-f56491d63297" (UID: "9b2a359d-5ea3-4193-8c83-f56491d63297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:11.535473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.535423 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz" pod "9b2a359d-5ea3-4193-8c83-f56491d63297" (UID: "9b2a359d-5ea3-4193-8c83-f56491d63297"). InnerVolumeSpecName "kube-api-access-c4bdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:11.628646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.626979 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4bdz\" (UniqueName: \"kubernetes.io/projected/9b2a359d-5ea3-4193-8c83-f56491d63297-kube-api-access-c4bdz\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:11.628646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:11.627014 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b2a359d-5ea3-4193-8c83-f56491d63297-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:12.188072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:12.188040 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" Nov 03 21:08:12.188072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:12.188064 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487" event={"ID":"9b2a359d-5ea3-4193-8c83-f56491d63297","Type":"ContainerDied","Data":"d715ecb09f7c28fa56a22a602b006af55b70d51ed3a291d561f714bd314880b8"} Nov 03 21:08:12.188317 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:12.188119 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d715ecb09f7c28fa56a22a602b006af55b70d51ed3a291d561f714bd314880b8" Nov 03 21:08:14.154370 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:14.154326 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:08:14.737358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:14.731828 5932 patch_prober.go:29] interesting pod/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:14.737358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:14.731892 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:15.088557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.088469 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:08:15.088557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.088537 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:08:15.480345 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.480316 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:08:15.593722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.593689 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.593931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.593860 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.593931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.593887 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.593931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.593917 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.594111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.593966 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.594180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.594104 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.594180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.594163 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.594291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.594217 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.594291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.594237 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.594397 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.594298 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token\") pod \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\" (UID: \"3071cf01-341c-4548-bdad-5b7b4f0b9d41\") " Nov 03 21:08:15.594457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.594394 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.595087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.595059 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.595087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.595089 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.595391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.595345 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:15.595485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.595441 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.595550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.595491 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.597405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.597351 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:15.597882 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.597851 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:15.598236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.598202 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "kube-api-access-gzq77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:15.598337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.598301 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token" pod "3071cf01-341c-4548-bdad-5b7b4f0b9d41" (UID: "3071cf01-341c-4548-bdad-5b7b4f0b9d41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:15.696897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696856 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.696897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696893 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzq77\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-kube-api-access-gzq77\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.696897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696902 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.697126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696911 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.697126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696919 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.697126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696929 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3071cf01-341c-4548-bdad-5b7b4f0b9d41-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:15.697126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:15.696938 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3071cf01-341c-4548-bdad-5b7b4f0b9d41-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:16.211586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.211549 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr"] Nov 03 21:08:16.211865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.211817 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" containerID="cri-o://165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c" gracePeriod=30 Nov 03 21:08:16.227382 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.227348 5932 generic.go:356] "Generic (PLEG): container finished" podID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerID="f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b" exitCode=0 Nov 03 21:08:16.227591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.227420 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" Nov 03 21:08:16.227591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.227428 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" event={"ID":"3071cf01-341c-4548-bdad-5b7b4f0b9d41","Type":"ContainerDied","Data":"f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b"} Nov 03 21:08:16.227591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.227500 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d" event={"ID":"3071cf01-341c-4548-bdad-5b7b4f0b9d41","Type":"ContainerDied","Data":"dc76f2770f67c8936bf67a24907fd4a3586fd131ba2698d39d26f2778900ee7d"} Nov 03 21:08:16.227591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.227532 5932 scope.go:122] "RemoveContainer" containerID="f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b" Nov 03 21:08:16.240825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.240801 5932 scope.go:122] "RemoveContainer" containerID="f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387" Nov 03 21:08:16.251787 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.251765 5932 scope.go:122] "RemoveContainer" containerID="f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b" Nov 03 21:08:16.252083 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:16.252060 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b\": container with ID starting with f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b not found: ID does not exist" containerID="f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b" Nov 03 21:08:16.252168 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.252096 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b"} err="failed to get container status \"f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b\": rpc error: code = NotFound desc = could not find container \"f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b\": container with ID starting with f328c5394a10929f5bce9761161533ae8627c596faa4a717e4d66a8fdddb637b not found: ID does not exist" Nov 03 21:08:16.252168 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.252123 5932 scope.go:122] "RemoveContainer" containerID="f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387" Nov 03 21:08:16.252385 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:16.252367 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387\": container with ID starting with f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387 not found: ID does not exist" containerID="f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387" Nov 03 21:08:16.252427 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.252394 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387"} err="failed to get container status \"f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387\": rpc error: code = NotFound desc = could not find container \"f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387\": container with ID starting with f93cec05baf48e3b1c1cc5f6ca6020d2554096dfe4e05e6c092c2204cf420387 not found: ID does not exist" Nov 03 21:08:16.469467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.467766 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d"] Nov 03 21:08:16.473898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:16.473867 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0776c6-openshift-default-b684c75c7-fs55d"] Nov 03 21:08:17.040436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.040405 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj"] Nov 03 21:08:17.041193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041175 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9b2a359d-5ea3-4193-8c83-f56491d63297" containerName="revoke-keys" Nov 03 21:08:17.041193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041193 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2a359d-5ea3-4193-8c83-f56491d63297" containerName="revoke-keys" Nov 03 21:08:17.041316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041204 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" Nov 03 21:08:17.041316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041210 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" Nov 03 21:08:17.041316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041219 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" Nov 03 21:08:17.041316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041224 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" Nov 03 21:08:17.041518 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041350 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" Nov 03 21:08:17.041518 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.041371 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2a359d-5ea3-4193-8c83-f56491d63297" containerName="revoke-keys" Nov 03 21:08:17.046652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.046578 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.100084 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.100049 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj"] Nov 03 21:08:17.125370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.125334 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:17.125597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.125408 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:08:17.126060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.126028 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.126237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.126214 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.126431 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.126398 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztkg\" (UniqueName: \"kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.228445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.228410 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.228705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.228505 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.228705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.228554 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztkg\" (UniqueName: \"kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.229366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.229340 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.235095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.235058 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.290351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.290320 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztkg\" (UniqueName: \"kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.362358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.362275 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:17.748310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:17.748194 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj"] Nov 03 21:08:17.752818 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:17.752787 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb702dc3_5fc1_4e2c_bfcb_eb88bf801b47.slice/crio-5930944325c96ce4e7566229f3590f988639f54465c7635e16a8b7f99593586a WatchSource:0}: Error finding container 5930944325c96ce4e7566229f3590f988639f54465c7635e16a8b7f99593586a: Status 404 returned error can't find the container with id 5930944325c96ce4e7566229f3590f988639f54465c7635e16a8b7f99593586a Nov 03 21:08:18.053732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:18.053500 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" path="/var/lib/kubelet/pods/3071cf01-341c-4548-bdad-5b7b4f0b9d41/volumes" Nov 03 21:08:18.248806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:18.248771 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" event={"ID":"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47","Type":"ContainerStarted","Data":"2edfdd7bbecaad2b99786725d4846ab95d0fbcd1e9678e022aa0f15f281ed7b8"} Nov 03 21:08:18.248965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:18.248826 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" event={"ID":"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47","Type":"ContainerStarted","Data":"5930944325c96ce4e7566229f3590f988639f54465c7635e16a8b7f99593586a"} Nov 03 21:08:18.358514 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:18.358391 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" podStartSLOduration=2.358376213 podStartE2EDuration="2.358376213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:18.357294801 +0000 UTC m=+2306.422536275" watchObservedRunningTime="2026-11-03 21:08:18.358376213 +0000 UTC m=+2306.423617686" Nov 03 21:08:18.715011 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:18.714983 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb702dc3_5fc1_4e2c_bfcb_eb88bf801b47.slice/crio-conmon-2edfdd7bbecaad2b99786725d4846ab95d0fbcd1e9678e022aa0f15f281ed7b8.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:08:18.715132 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:18.715013 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb702dc3_5fc1_4e2c_bfcb_eb88bf801b47.slice/crio-2edfdd7bbecaad2b99786725d4846ab95d0fbcd1e9678e022aa0f15f281ed7b8.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:08:19.262549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:19.262507 5932 generic.go:356] "Generic (PLEG): container finished" podID="db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" containerID="2edfdd7bbecaad2b99786725d4846ab95d0fbcd1e9678e022aa0f15f281ed7b8" exitCode=0 Nov 03 21:08:19.262992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:19.262649 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" event={"ID":"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47","Type":"ContainerDied","Data":"2edfdd7bbecaad2b99786725d4846ab95d0fbcd1e9678e022aa0f15f281ed7b8"} Nov 03 21:08:20.520266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.520239 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:20.586797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.586703 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca\") pod \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " Nov 03 21:08:20.586797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.586746 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca\") pod \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " Nov 03 21:08:20.587046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.586814 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg\" (UniqueName: \"kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg\") pod \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\" (UID: \"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47\") " Nov 03 21:08:20.587143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.587101 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca" pod "db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" (UID: "db702dc3-5fc1-4e2c-bfcb-eb88bf801b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:20.587274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.587258 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:20.589076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.589047 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg" pod "db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" (UID: "db702dc3-5fc1-4e2c-bfcb-eb88bf801b47"). InnerVolumeSpecName "kube-api-access-dztkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:20.589182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.589087 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca" pod "db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" (UID: "db702dc3-5fc1-4e2c-bfcb-eb88bf801b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:20.688262 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.688225 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:20.688262 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:20.688260 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztkg\" (UniqueName: \"kubernetes.io/projected/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47-kube-api-access-dztkg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.365878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.365851 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" Nov 03 21:08:21.366449 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.365719 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj" event={"ID":"db702dc3-5fc1-4e2c-bfcb-eb88bf801b47","Type":"ContainerDied","Data":"5930944325c96ce4e7566229f3590f988639f54465c7635e16a8b7f99593586a"} Nov 03 21:08:21.366596 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.366491 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5930944325c96ce4e7566229f3590f988639f54465c7635e16a8b7f99593586a" Nov 03 21:08:21.828459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.828431 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 21:08:21.934415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934377 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934429 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934526 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934651 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934680 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934713 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934772 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.934894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934851 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.935134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.934900 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data\") pod \"2314c3a7-3dbe-4574-9c32-42175696ec7c\" (UID: \"2314c3a7-3dbe-4574-9c32-42175696ec7c\") " Nov 03 21:08:21.935558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.935451 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.935766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.935686 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.935766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.935705 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.935766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.935742 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:21.935989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.935955 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.936157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.936126 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:21.938138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.938104 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:21.938973 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.938940 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:21.939323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.939290 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "kube-api-access-hct9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:21.939435 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:21.939422 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo" pod "2314c3a7-3dbe-4574-9c32-42175696ec7c" (UID: "2314c3a7-3dbe-4574-9c32-42175696ec7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:22.037277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037227 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037276 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037291 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037306 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037320 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037333 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2314c3a7-3dbe-4574-9c32-42175696ec7c-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037348 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hct9c\" (UniqueName: \"kubernetes.io/projected/2314c3a7-3dbe-4574-9c32-42175696ec7c-kube-api-access-hct9c\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.037539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.037363 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2314c3a7-3dbe-4574-9c32-42175696ec7c-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:22.333662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.333548 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v"] Nov 03 21:08:22.376203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.376167 5932 generic.go:356] "Generic (PLEG): container finished" podID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerID="165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c" exitCode=0 Nov 03 21:08:22.376388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.376228 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" Nov 03 21:08:22.376388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.376256 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" event={"ID":"2314c3a7-3dbe-4574-9c32-42175696ec7c","Type":"ContainerDied","Data":"165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c"} Nov 03 21:08:22.376388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.376328 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr" event={"ID":"2314c3a7-3dbe-4574-9c32-42175696ec7c","Type":"ContainerDied","Data":"51ab4b9f4302bdec341717a35cc26a9e4bbd55665df72a3756e8af77ececd9cb"} Nov 03 21:08:22.376388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.376357 5932 scope.go:122] "RemoveContainer" containerID="165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c" Nov 03 21:08:22.388572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.388542 5932 scope.go:122] "RemoveContainer" containerID="66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387" Nov 03 21:08:22.399806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.399780 5932 scope.go:122] "RemoveContainer" containerID="165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c" Nov 03 21:08:22.400275 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:22.400246 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c\": container with ID starting with 165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c not found: ID does not exist" containerID="165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c" Nov 03 21:08:22.400385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.400284 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c"} err="failed to get container status \"165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c\": rpc error: code = NotFound desc = could not find container \"165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c\": container with ID starting with 165863731aa60e14f3de96cb073368e2a77ed121e279331a73b6f8a78ade4e7c not found: ID does not exist" Nov 03 21:08:22.400385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.400313 5932 scope.go:122] "RemoveContainer" containerID="66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387" Nov 03 21:08:22.400685 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:22.400663 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387\": container with ID starting with 66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387 not found: ID does not exist" containerID="66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387" Nov 03 21:08:22.400760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.400693 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387"} err="failed to get container status \"66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387\": rpc error: code = NotFound desc = could not find container \"66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387\": container with ID starting with 66569525731b8f34772f520d628353c57c72c33ed916c5b5ce52374c448c9387 not found: ID does not exist" Nov 03 21:08:22.447954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.445724 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw"] Nov 03 21:08:22.665988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.665770 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr"] Nov 03 21:08:22.670593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:22.670558 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-cc5d0f-openshift-default-5488658cff-tsfxr"] Nov 03 21:08:24.112647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:24.111287 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" path="/var/lib/kubelet/pods/2314c3a7-3dbe-4574-9c32-42175696ec7c/volumes" Nov 03 21:08:27.334051 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:27.334007 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerName="payload-processing" containerID="cri-o://bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc" gracePeriod=25 Nov 03 21:08:27.446098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:27.446053 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerName="payload-pre-processing" containerID="cri-o://248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a" gracePeriod=25 Nov 03 21:08:30.290313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.290277 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf"] Nov 03 21:08:30.291048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291031 5932 cpu_manager.go:414] "containerMap: removing container" podUID="db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" containerName="revoke-keys" Nov 03 21:08:30.291101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291048 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" containerName="revoke-keys" Nov 03 21:08:30.291101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291082 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" Nov 03 21:08:30.291101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291087 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" Nov 03 21:08:30.291101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291097 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" Nov 03 21:08:30.291230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291104 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" Nov 03 21:08:30.291270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291232 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" containerName="revoke-keys" Nov 03 21:08:30.291270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291239 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3071cf01-341c-4548-bdad-5b7b4f0b9d41" containerName="istio-proxy" Nov 03 21:08:30.291270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291250 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" Nov 03 21:08:30.291270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.291256 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2314c3a7-3dbe-4574-9c32-42175696ec7c" containerName="istio-proxy" Nov 03 21:08:30.296704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.296684 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.352123 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.352079 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf"] Nov 03 21:08:30.391594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.391525 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7a3593-serving-cert\"" Nov 03 21:08:30.406162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.406121 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.406370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.406266 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.406370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.406368 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2hw\" (UniqueName: \"kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.406539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.406519 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.507832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.507789 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.508060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.507896 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.508060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.508010 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2hw\" (UniqueName: \"kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.508186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.508130 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.508292 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:30.508264 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-serving-cert: secret "maas-api-e2e-api-a-7a3593-serving-cert" not found Nov 03 21:08:30.508360 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:30.508348 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls podName:735ea8bc-59ec-4b56-b1cb-fad48cc98675 nodeName:}" failed. No retries permitted until 2026-11-03 21:08:31.008324485 +0000 UTC m=+2319.073565940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls") pod "maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" (UID: "735ea8bc-59ec-4b56-b1cb-fad48cc98675") : secret "maas-api-e2e-api-a-7a3593-serving-cert" not found Nov 03 21:08:30.508610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.508591 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.510415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.510389 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:30.594280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:30.594136 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2hw\" (UniqueName: \"kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:31.016741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:31.016702 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:31.019443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:31.019412 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\") pod \"maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:31.213055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:31.213016 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:31.624304 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:31.624276 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf"] Nov 03 21:08:31.628874 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:31.628841 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735ea8bc_59ec_4b56_b1cb_fad48cc98675.slice/crio-491d33261c3a4b199dc278bf2662ab28cf1263ff1cdc00fbaa3895a2ebe6d86e WatchSource:0}: Error finding container 491d33261c3a4b199dc278bf2662ab28cf1263ff1cdc00fbaa3895a2ebe6d86e: Status 404 returned error can't find the container with id 491d33261c3a4b199dc278bf2662ab28cf1263ff1cdc00fbaa3895a2ebe6d86e Nov 03 21:08:32.467322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:32.467287 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" event={"ID":"735ea8bc-59ec-4b56-b1cb-fad48cc98675","Type":"ContainerStarted","Data":"491d33261c3a4b199dc278bf2662ab28cf1263ff1cdc00fbaa3895a2ebe6d86e"} Nov 03 21:08:33.590024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:33.580687 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" event={"ID":"735ea8bc-59ec-4b56-b1cb-fad48cc98675","Type":"ContainerStarted","Data":"73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce"} Nov 03 21:08:33.590024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:33.582506 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:33.701288 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:33.701231 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" podStartSLOduration=2.76924299 podStartE2EDuration="3.701215271s" totalImagesPullingTime="931.972281ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:33.700158575 +0000 UTC m=+2321.765400071" watchObservedRunningTime="2026-11-03 21:08:33.701215271 +0000 UTC m=+2321.766456744" Nov 03 21:08:35.116212 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:35.116153 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:08:35.140651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:35.133804 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:08:40.369828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.369786 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6"] Nov 03 21:08:40.378510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.378474 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.424176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.424142 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6"] Nov 03 21:08:40.434843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.434808 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslvz\" (UniqueName: \"kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.435095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.435067 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.435184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.435170 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.469803 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.469767 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:08:40.470062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.469791 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:08:40.536431 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.536392 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.536663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.536452 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.536663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.536540 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslvz\" (UniqueName: \"kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.537212 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.537188 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.538977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.538953 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.594104 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.594068 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslvz\" (UniqueName: \"kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz\") pod \"maas-api-revoke-sub-e76ad77a-vtzt6\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:40.608152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.608122 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:08:40.696267 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:40.696225 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:41.099225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:41.099036 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6"] Nov 03 21:08:41.103958 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:08:41.103925 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389f9a4c_7b7a_44c6_9ac6_c1c93fbdd764.slice/crio-ba59431b69df1f188c6035d2dcff1e634a32aeb881a46fdde449265c50c4ab67 WatchSource:0}: Error finding container ba59431b69df1f188c6035d2dcff1e634a32aeb881a46fdde449265c50c4ab67: Status 404 returned error can't find the container with id ba59431b69df1f188c6035d2dcff1e634a32aeb881a46fdde449265c50c4ab67 Nov 03 21:08:41.727138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:41.727097 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" event={"ID":"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764","Type":"ContainerStarted","Data":"3ed4ac63b3c0ec582b8e689c0522e0205abb467c7fef3c1e78ee566f053bedc7"} Nov 03 21:08:41.727528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:41.727162 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" event={"ID":"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764","Type":"ContainerStarted","Data":"ba59431b69df1f188c6035d2dcff1e634a32aeb881a46fdde449265c50c4ab67"} Nov 03 21:08:41.837795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:41.837685 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" podStartSLOduration=1.8376674830000002 podStartE2EDuration="1.837667483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:08:41.834969139 +0000 UTC m=+2329.900210612" watchObservedRunningTime="2026-11-03 21:08:41.837667483 +0000 UTC m=+2329.902908955" Nov 03 21:08:42.738305 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:42.738269 5932 generic.go:356] "Generic (PLEG): container finished" podID="389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" containerID="3ed4ac63b3c0ec582b8e689c0522e0205abb467c7fef3c1e78ee566f053bedc7" exitCode=0 Nov 03 21:08:42.738788 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:42.738362 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" event={"ID":"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764","Type":"ContainerDied","Data":"3ed4ac63b3c0ec582b8e689c0522e0205abb467c7fef3c1e78ee566f053bedc7"} Nov 03 21:08:43.920144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.920116 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:43.996014 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.995982 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca\") pod \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " Nov 03 21:08:43.996187 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.996173 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz\" (UniqueName: \"kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz\") pod \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " Nov 03 21:08:43.996267 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.996231 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca\") pod \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\" (UID: \"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764\") " Nov 03 21:08:43.996388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.996361 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca" pod "389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" (UID: "389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:43.996801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.996762 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:43.998354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.998301 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca" pod "389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" (UID: "389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:43.998451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:43.998374 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz" pod "389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" (UID: "389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764"). InnerVolumeSpecName "kube-api-access-jslvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:44.098680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:44.098642 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jslvz\" (UniqueName: \"kubernetes.io/projected/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-kube-api-access-jslvz\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:44.098680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:44.098682 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:44.157984 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:44.157945 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:08:44.756078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:44.756049 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" Nov 03 21:08:44.756268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:44.756087 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6" event={"ID":"389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764","Type":"ContainerDied","Data":"ba59431b69df1f188c6035d2dcff1e634a32aeb881a46fdde449265c50c4ab67"} Nov 03 21:08:44.756268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:44.756163 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba59431b69df1f188c6035d2dcff1e634a32aeb881a46fdde449265c50c4ab67" Nov 03 21:08:45.092461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:45.092369 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:08:45.092896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:45.092369 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:08:46.368372 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:46.368329 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn"] Nov 03 21:08:46.368915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:46.368613 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" containerID="cri-o://d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4" gracePeriod=30 Nov 03 21:08:46.470340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:46.470300 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl"] Nov 03 21:08:46.470604 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:46.470569 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" containerID="cri-o://93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103" gracePeriod=30 Nov 03 21:08:46.998706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:46.998669 5932 patch_prober.go:29] interesting pod/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:08:46.998915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:46.998731 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:08:47.620104 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.620079 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 21:08:47.741237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.741203 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 21:08:47.759666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.759610 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb\" (UniqueName: \"kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb\") pod \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " Nov 03 21:08:47.759865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.759680 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume\") pod \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\" (UID: \"3bee0c30-a7ba-4da5-8c82-741760c6a2f9\") " Nov 03 21:08:47.760114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.760083 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume" pod "3bee0c30-a7ba-4da5-8c82-741760c6a2f9" (UID: "3bee0c30-a7ba-4da5-8c82-741760c6a2f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:47.760473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.760442 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:47.762097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.762066 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb" pod "3bee0c30-a7ba-4da5-8c82-741760c6a2f9" (UID: "3bee0c30-a7ba-4da5-8c82-741760c6a2f9"). InnerVolumeSpecName "kube-api-access-bzntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:47.783657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.783594 5932 generic.go:356] "Generic (PLEG): container finished" podID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerID="bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc" exitCode=0 Nov 03 21:08:47.783657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.783648 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" event={"ID":"3bee0c30-a7ba-4da5-8c82-741760c6a2f9","Type":"ContainerDied","Data":"bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc"} Nov 03 21:08:47.783853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.783678 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" Nov 03 21:08:47.783853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.783700 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v" event={"ID":"3bee0c30-a7ba-4da5-8c82-741760c6a2f9","Type":"ContainerDied","Data":"39211b406ca29b8ae644050b4cc1d0108d84175ca34e8bfd2c6b891cb3aabafa"} Nov 03 21:08:47.783853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.783726 5932 scope.go:122] "RemoveContainer" containerID="bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc" Nov 03 21:08:47.785450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.785427 5932 generic.go:356] "Generic (PLEG): container finished" podID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerID="248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a" exitCode=0 Nov 03 21:08:47.785559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.785474 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" Nov 03 21:08:47.785559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.785506 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" event={"ID":"3647f3e8-3643-4845-b8ac-b7577b04ba1a","Type":"ContainerDied","Data":"248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a"} Nov 03 21:08:47.785559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.785555 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw" event={"ID":"3647f3e8-3643-4845-b8ac-b7577b04ba1a","Type":"ContainerDied","Data":"6990ca051ca0da46fcacaede7998d89357fc0be7e1f749afa432063bcd11bfa1"} Nov 03 21:08:47.799687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.796761 5932 scope.go:122] "RemoveContainer" containerID="bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc" Nov 03 21:08:47.799859 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:47.799833 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc\": container with ID starting with bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc not found: ID does not exist" containerID="bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc" Nov 03 21:08:47.799911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.799871 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc"} err="failed to get container status \"bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc\": rpc error: code = NotFound desc = could not find container \"bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc\": container with ID starting with bb9c0c6416b78e9ca5e1ee16a978dee6604db9de3cde9bd1ff3cd24fb541d0bc not found: ID does not exist" Nov 03 21:08:47.799911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.799895 5932 scope.go:122] "RemoveContainer" containerID="248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a" Nov 03 21:08:47.810494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.810472 5932 scope.go:122] "RemoveContainer" containerID="248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a" Nov 03 21:08:47.810828 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:47.810807 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a\": container with ID starting with 248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a not found: ID does not exist" containerID="248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a" Nov 03 21:08:47.810900 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.810837 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a"} err="failed to get container status \"248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a\": rpc error: code = NotFound desc = could not find container \"248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a\": container with ID starting with 248de66250c2cd83890f8d70eed22108e6895db1d31ed6542ac6c9cb5459882a not found: ID does not exist" Nov 03 21:08:47.866854 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.866766 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume\") pod \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " Nov 03 21:08:47.867006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.866868 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c\" (UniqueName: \"kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c\") pod \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\" (UID: \"3647f3e8-3643-4845-b8ac-b7577b04ba1a\") " Nov 03 21:08:47.867810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.867788 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzntb\" (UniqueName: \"kubernetes.io/projected/3bee0c30-a7ba-4da5-8c82-741760c6a2f9-kube-api-access-bzntb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:47.868714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.868677 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume" pod "3647f3e8-3643-4845-b8ac-b7577b04ba1a" (UID: "3647f3e8-3643-4845-b8ac-b7577b04ba1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:47.870192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.870165 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c" pod "3647f3e8-3643-4845-b8ac-b7577b04ba1a" (UID: "3647f3e8-3643-4845-b8ac-b7577b04ba1a"). InnerVolumeSpecName "kube-api-access-gb46c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:47.957544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.957503 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v"] Nov 03 21:08:47.965907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.965863 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3d2008-766f7cffcc-lbk7v"] Nov 03 21:08:47.968720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.968686 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3647f3e8-3643-4845-b8ac-b7577b04ba1a-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:47.968720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:47.968712 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb46c\" (UniqueName: \"kubernetes.io/projected/3647f3e8-3643-4845-b8ac-b7577b04ba1a-kube-api-access-gb46c\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:48.053978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:48.053933 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" path="/var/lib/kubelet/pods/3bee0c30-a7ba-4da5-8c82-741760c6a2f9/volumes" Nov 03 21:08:48.054637 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:48.054603 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn"] Nov 03 21:08:48.054758 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:48.054657 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8faf882d-4w9bn"] Nov 03 21:08:48.197551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:48.197514 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw"] Nov 03 21:08:48.201307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:48.201281 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3d2008-68f86c99dc-75fsw"] Nov 03 21:08:50.053006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.052974 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19154b12-c9d0-418c-8963-35a9ee86efb2" path="/var/lib/kubelet/pods/19154b12-c9d0-418c-8963-35a9ee86efb2/volumes" Nov 03 21:08:50.053454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.053439 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" path="/var/lib/kubelet/pods/3647f3e8-3643-4845-b8ac-b7577b04ba1a/volumes" Nov 03 21:08:50.370440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.370356 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57"] Nov 03 21:08:50.370614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.370558 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" podUID="3e1d8531-44c6-4f83-a161-2be10803677b" containerName="maas-api" containerID="cri-o://94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0" gracePeriod=30 Nov 03 21:08:50.724733 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.724698 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:08:50.828676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.827535 5932 generic.go:356] "Generic (PLEG): container finished" podID="3e1d8531-44c6-4f83-a161-2be10803677b" containerID="94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0" exitCode=0 Nov 03 21:08:50.828676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.827641 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" event={"ID":"3e1d8531-44c6-4f83-a161-2be10803677b","Type":"ContainerDied","Data":"94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0"} Nov 03 21:08:50.828676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.827692 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" event={"ID":"3e1d8531-44c6-4f83-a161-2be10803677b","Type":"ContainerDied","Data":"a339246a77caaff3a61bf59d8c17b2ac6b7d465db776b1ffa72c52897d7d099b"} Nov 03 21:08:50.828676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.827724 5932 scope.go:122] "RemoveContainer" containerID="94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0" Nov 03 21:08:50.828676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.827892 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57" Nov 03 21:08:50.843002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.842970 5932 scope.go:122] "RemoveContainer" containerID="94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0" Nov 03 21:08:50.843374 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:50.843350 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0\": container with ID starting with 94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0 not found: ID does not exist" containerID="94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0" Nov 03 21:08:50.843450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.843384 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0"} err="failed to get container status \"94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0\": rpc error: code = NotFound desc = could not find container \"94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0\": container with ID starting with 94371ddd2cae8f48b0d67420710f32c4927e87190b5bed0072157ee461fb37c0 not found: ID does not exist" Nov 03 21:08:50.886643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.881999 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\") pod \"3e1d8531-44c6-4f83-a161-2be10803677b\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " Nov 03 21:08:50.886643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.882061 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs\") pod \"3e1d8531-44c6-4f83-a161-2be10803677b\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " Nov 03 21:08:50.886643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.882131 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca\") pod \"3e1d8531-44c6-4f83-a161-2be10803677b\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " Nov 03 21:08:50.886643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.882272 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg\" (UniqueName: \"kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg\") pod \"3e1d8531-44c6-4f83-a161-2be10803677b\" (UID: \"3e1d8531-44c6-4f83-a161-2be10803677b\") " Nov 03 21:08:50.886643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.884552 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca" pod "3e1d8531-44c6-4f83-a161-2be10803677b" (UID: "3e1d8531-44c6-4f83-a161-2be10803677b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:50.901641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.899516 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls" pod "3e1d8531-44c6-4f83-a161-2be10803677b" (UID: "3e1d8531-44c6-4f83-a161-2be10803677b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:50.909509 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.909468 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg" pod "3e1d8531-44c6-4f83-a161-2be10803677b" (UID: "3e1d8531-44c6-4f83-a161-2be10803677b"). InnerVolumeSpecName "kube-api-access-nbfcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:50.909742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.909579 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs" pod "3e1d8531-44c6-4f83-a161-2be10803677b" (UID: "3e1d8531-44c6-4f83-a161-2be10803677b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:08:50.985250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.985164 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfcg\" (UniqueName: \"kubernetes.io/projected/3e1d8531-44c6-4f83-a161-2be10803677b-kube-api-access-nbfcg\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:50.985250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.985196 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:50.985250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.985207 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e1d8531-44c6-4f83-a161-2be10803677b-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:50.985250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:50.985216 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e1d8531-44c6-4f83-a161-2be10803677b-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:51.252538 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.252439 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57"] Nov 03 21:08:51.296792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.296758 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e68f8112-7598458dbc-sfq57"] Nov 03 21:08:51.445943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.443242 5932 patch_prober.go:29] interesting pod/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:55512->10.132.0.60:15021: read: connection reset by peer" start-of-body= Nov 03 21:08:51.445943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.443304 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": read tcp 10.132.0.2:55512->10.132.0.60:15021: read: connection reset by peer" Nov 03 21:08:51.752975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.752946 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 21:08:51.775185 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.775159 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 21:08:51.845043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.845000 5932 generic.go:356] "Generic (PLEG): container finished" podID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerID="93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103" exitCode=0 Nov 03 21:08:51.845247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.845164 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" event={"ID":"3bec572b-4561-494a-abdc-11a7e4b2f7d3","Type":"ContainerDied","Data":"93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103"} Nov 03 21:08:51.845326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.845252 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" event={"ID":"3bec572b-4561-494a-abdc-11a7e4b2f7d3","Type":"ContainerDied","Data":"63ba695460e54495e2e99ae10bd82ca51b22a4564c05865dc100e827a9dd845d"} Nov 03 21:08:51.845326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.845296 5932 scope.go:122] "RemoveContainer" containerID="93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103" Nov 03 21:08:51.846129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.846099 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl" Nov 03 21:08:51.849109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.849067 5932 generic.go:356] "Generic (PLEG): container finished" podID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerID="d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4" exitCode=0 Nov 03 21:08:51.849366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.849345 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" event={"ID":"051ea739-77c3-43ae-bcda-7d1915deeb20","Type":"ContainerDied","Data":"d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4"} Nov 03 21:08:51.849439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.849409 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" event={"ID":"051ea739-77c3-43ae-bcda-7d1915deeb20","Type":"ContainerDied","Data":"3aa9a33f290cea07db93e9f5969212cc44eb1359fe0bf1e844f7293ce50b15fd"} Nov 03 21:08:51.849547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.849526 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn" Nov 03 21:08:51.864369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.864336 5932 scope.go:122] "RemoveContainer" containerID="93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103" Nov 03 21:08:51.864801 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:51.864778 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103\": container with ID starting with 93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103 not found: ID does not exist" containerID="93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103" Nov 03 21:08:51.864915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.864811 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103"} err="failed to get container status \"93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103\": rpc error: code = NotFound desc = could not find container \"93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103\": container with ID starting with 93b74f6599e6be9056a62ee81c47541857f766a7e1be0a119a975c8e30011103 not found: ID does not exist" Nov 03 21:08:51.864915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.864831 5932 scope.go:122] "RemoveContainer" containerID="d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4" Nov 03 21:08:51.881095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.881067 5932 scope.go:122] "RemoveContainer" containerID="0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e" Nov 03 21:08:51.902820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.902793 5932 scope.go:122] "RemoveContainer" containerID="d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4" Nov 03 21:08:51.903262 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903237 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.903401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903277 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.903401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903309 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.903401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903360 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903420 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903446 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903485 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903502 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:51.903518 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4\": container with ID starting with d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4 not found: ID does not exist" containerID="d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4" Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903537 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903563 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.903575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903559 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4"} err="failed to get container status \"d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4\": rpc error: code = NotFound desc = could not find container \"d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4\": container with ID starting with d2f9e9e6a0f4b566ad13d0ccb6978fbe17bc6c3e96293b1c93f9dd05f4ad44c4 not found: ID does not exist" Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903585 5932 scope.go:122] "RemoveContainer" containerID="0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e" Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903586 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903792 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903832 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903864 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903923 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket\") pod \"051ea739-77c3-43ae-bcda-7d1915deeb20\" (UID: \"051ea739-77c3-43ae-bcda-7d1915deeb20\") " Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.903982 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.904070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.904014 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.904508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.904075 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data\") pod \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\" (UID: \"3bec572b-4561-494a-abdc-11a7e4b2f7d3\") " Nov 03 21:08:51.905150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.905125 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.906773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.905718 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:51.906773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.905988 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.907678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.906261 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:51.909613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.909589 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "kube-api-access-h8ckq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:51.909842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.909823 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.912150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.912121 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.912150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.912133 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.912381 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.912332 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.912536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.912515 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.912536 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:08:51.912521 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e\": container with ID starting with 0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e not found: ID does not exist" containerID="0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e" Nov 03 21:08:51.912722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.912521 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.912722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.912554 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e"} err="failed to get container status \"0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e\": rpc error: code = NotFound desc = could not find container \"0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e\": container with ID starting with 0cbefad31c13dce16aa6c823cd1adafe3e7beb5adeb85992dd2edaba7200210e not found: ID does not exist" Nov 03 21:08:51.919760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.913000 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:08:51.919760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.913234 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.919760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.918414 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "kube-api-access-hrznl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:51.919760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.918708 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:08:51.919760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.918858 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo" pod "051ea739-77c3-43ae-bcda-7d1915deeb20" (UID: "051ea739-77c3-43ae-bcda-7d1915deeb20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:51.921043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.921011 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:08:51.929144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:51.929111 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token" pod "3bec572b-4561-494a-abdc-11a7e4b2f7d3" (UID: "3bec572b-4561-494a-abdc-11a7e4b2f7d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:08:52.006158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006122 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006157 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006167 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006176 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006186 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006194 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006202 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006211 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/051ea739-77c3-43ae-bcda-7d1915deeb20-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006219 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ckq\" (UniqueName: \"kubernetes.io/projected/3bec572b-4561-494a-abdc-11a7e4b2f7d3-kube-api-access-h8ckq\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006229 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006238 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006247 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006258 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec572b-4561-494a-abdc-11a7e4b2f7d3-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006266 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrznl\" (UniqueName: \"kubernetes.io/projected/051ea739-77c3-43ae-bcda-7d1915deeb20-kube-api-access-hrznl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006275 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006283 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/051ea739-77c3-43ae-bcda-7d1915deeb20-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006291 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.006412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.006301 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec572b-4561-494a-abdc-11a7e4b2f7d3-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:08:52.066387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.066268 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e1d8531-44c6-4f83-a161-2be10803677b" path="/var/lib/kubelet/pods/3e1d8531-44c6-4f83-a161-2be10803677b/volumes" Nov 03 21:08:52.286388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.286350 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn"] Nov 03 21:08:52.292867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.292837 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3d2008-openshift-default-587f586467-fsbhn"] Nov 03 21:08:52.396402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.396367 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl"] Nov 03 21:08:52.406365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.406333 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-504eb0-openshift-default-5c944f6b6f-sklrl"] Nov 03 21:08:52.716874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.716775 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487"] Nov 03 21:08:52.723822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:52.723781 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e68f8112-p8487"] Nov 03 21:08:54.054528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:54.054487 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" path="/var/lib/kubelet/pods/051ea739-77c3-43ae-bcda-7d1915deeb20/volumes" Nov 03 21:08:54.055084 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:54.055067 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" path="/var/lib/kubelet/pods/3bec572b-4561-494a-abdc-11a7e4b2f7d3/volumes" Nov 03 21:08:54.055492 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:54.055478 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2a359d-5ea3-4193-8c83-f56491d63297" path="/var/lib/kubelet/pods/9b2a359d-5ea3-4193-8c83-f56491d63297/volumes" Nov 03 21:08:58.068581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:58.068539 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk"] Nov 03 21:08:59.800062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:08:59.800025 5932 scope.go:122] "RemoveContainer" containerID="31de8b45c43eabba33aac1f10b282dabba5020c52ff1fbd42c21984658187a5a" Nov 03 21:09:07.449092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.448995 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5"] Nov 03 21:09:07.451999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.451966 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerName="payload-pre-processing" Nov 03 21:09:07.451999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.451996 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerName="payload-pre-processing" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452017 5932 cpu_manager.go:414] "containerMap: removing container" podUID="389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" containerName="revoke-keys" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452024 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" containerName="revoke-keys" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452036 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerName="payload-processing" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452043 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerName="payload-processing" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452062 5932 cpu_manager.go:414] "containerMap: removing container" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452069 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452082 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3e1d8531-44c6-4f83-a161-2be10803677b" containerName="maas-api" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452088 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e1d8531-44c6-4f83-a161-2be10803677b" containerName="maas-api" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452101 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452107 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452144 5932 cpu_manager.go:414] "containerMap: removing container" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" Nov 03 21:09:07.452215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452151 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" Nov 03 21:09:07.452868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452330 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3647f3e8-3643-4845-b8ac-b7577b04ba1a" containerName="payload-pre-processing" Nov 03 21:09:07.452868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452342 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" containerName="revoke-keys" Nov 03 21:09:07.452868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452353 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bee0c30-a7ba-4da5-8c82-741760c6a2f9" containerName="payload-processing" Nov 03 21:09:07.452868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452366 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" Nov 03 21:09:07.452868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452374 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e1d8531-44c6-4f83-a161-2be10803677b" containerName="maas-api" Nov 03 21:09:07.452868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.452383 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bec572b-4561-494a-abdc-11a7e4b2f7d3" containerName="istio-proxy" Nov 03 21:09:07.459450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.459423 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.483661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.476373 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5"] Nov 03 21:09:07.628645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.628062 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fc4e8d-serving-cert\"" Nov 03 21:09:07.637993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.631606 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.637993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.631777 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhhs\" (UniqueName: \"kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.637993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.631827 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.637993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.631870 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.733709 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.733598 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.733924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.733827 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.734004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.733937 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhhs\" (UniqueName: \"kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.734004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.733986 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.734779 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.734754 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.736313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.736288 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.736411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.736381 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.791406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.791374 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhhs\" (UniqueName: \"kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs\") pod \"maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:07.938651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:07.938087 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:08.404646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:08.404289 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5"] Nov 03 21:09:09.042645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:09.040530 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" event={"ID":"777f4471-9570-46a2-a547-35ab9f189bc5","Type":"ContainerStarted","Data":"165f498cde804ad1388dc9d74e19d617f143b9e8c0a8f9ef1151bf290f1cb628"} Nov 03 21:09:10.058270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:10.058235 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:10.058746 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:10.058297 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" event={"ID":"777f4471-9570-46a2-a547-35ab9f189bc5","Type":"ContainerStarted","Data":"5711ba510e7d0c8c4f9c9d4d5695e006995bb59f81d8dd1df17277072bce3822"} Nov 03 21:09:10.166536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:10.166484 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" podStartSLOduration=2.205428968 podStartE2EDuration="3.16647219s" totalImagesPullingTime="961.043222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:10.165842289 +0000 UTC m=+2358.231083763" watchObservedRunningTime="2026-11-03 21:09:10.16647219 +0000 UTC m=+2358.231713663" Nov 03 21:09:13.092061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.092001 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" containerID="cri-o://53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f" gracePeriod=45 Nov 03 21:09:13.451697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.451668 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 21:09:13.559130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559091 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm\") pod \"afb3b94d-a194-410a-aecf-9dfbff512cd2\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " Nov 03 21:09:13.559338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559166 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home\") pod \"afb3b94d-a194-410a-aecf-9dfbff512cd2\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " Nov 03 21:09:13.559338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559217 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd\") pod \"afb3b94d-a194-410a-aecf-9dfbff512cd2\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " Nov 03 21:09:13.559338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559250 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir\") pod \"afb3b94d-a194-410a-aecf-9dfbff512cd2\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " Nov 03 21:09:13.559338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559269 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs\") pod \"afb3b94d-a194-410a-aecf-9dfbff512cd2\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " Nov 03 21:09:13.559338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559302 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache\") pod \"afb3b94d-a194-410a-aecf-9dfbff512cd2\" (UID: \"afb3b94d-a194-410a-aecf-9dfbff512cd2\") " Nov 03 21:09:13.559810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559783 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache" pod "afb3b94d-a194-410a-aecf-9dfbff512cd2" (UID: "afb3b94d-a194-410a-aecf-9dfbff512cd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:13.559906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.559811 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home" pod "afb3b94d-a194-410a-aecf-9dfbff512cd2" (UID: "afb3b94d-a194-410a-aecf-9dfbff512cd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:13.560130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.560105 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir" pod "afb3b94d-a194-410a-aecf-9dfbff512cd2" (UID: "afb3b94d-a194-410a-aecf-9dfbff512cd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:13.561896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.561857 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs" pod "afb3b94d-a194-410a-aecf-9dfbff512cd2" (UID: "afb3b94d-a194-410a-aecf-9dfbff512cd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:13.562220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.562187 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm" pod "afb3b94d-a194-410a-aecf-9dfbff512cd2" (UID: "afb3b94d-a194-410a-aecf-9dfbff512cd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:13.563578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.563549 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd" pod "afb3b94d-a194-410a-aecf-9dfbff512cd2" (UID: "afb3b94d-a194-410a-aecf-9dfbff512cd2"). InnerVolumeSpecName "kube-api-access-s85cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:13.661041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.660955 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:13.661041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.660990 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85cd\" (UniqueName: \"kubernetes.io/projected/afb3b94d-a194-410a-aecf-9dfbff512cd2-kube-api-access-s85cd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:13.661041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.661000 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:13.661041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.661013 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afb3b94d-a194-410a-aecf-9dfbff512cd2-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:13.661041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.661021 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:13.661041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:13.661030 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afb3b94d-a194-410a-aecf-9dfbff512cd2-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:14.089702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.089664 5932 generic.go:356] "Generic (PLEG): container finished" podID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerID="53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f" exitCode=0 Nov 03 21:09:14.089907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.089757 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" Nov 03 21:09:14.089907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.089755 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" event={"ID":"afb3b94d-a194-410a-aecf-9dfbff512cd2","Type":"ContainerDied","Data":"53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f"} Nov 03 21:09:14.089907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.089870 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" event={"ID":"afb3b94d-a194-410a-aecf-9dfbff512cd2","Type":"ContainerDied","Data":"82a0614e672e538e6295635f00d9a093927037d0ad0741f159f0aafbbc4337ff"} Nov 03 21:09:14.089907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.089886 5932 scope.go:122] "RemoveContainer" containerID="53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f" Nov 03 21:09:14.101192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.101026 5932 scope.go:122] "RemoveContainer" containerID="53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f" Nov 03 21:09:14.101518 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:09:14.101428 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f\": container with ID starting with 53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f not found: ID does not exist" containerID="53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f" Nov 03 21:09:14.101518 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.101468 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f"} err="failed to get container status \"53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f\": rpc error: code = NotFound desc = could not find container \"53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f\": container with ID starting with 53a647884019dce5bb1d1f4bdadd4b85f58b480462cfb8a936fd6ff2fde60c5f not found: ID does not exist" Nov 03 21:09:14.180383 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:09:14.180345 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:09:14.252713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.252678 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk"] Nov 03 21:09:14.298453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.298422 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk"] Nov 03 21:09:14.315264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:14.315165 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-e68f8112/e2e-lifecycle-model-e68f8112-kserve-7958bdff4f-s2tbk" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.190:8000/ready\": context deadline exceeded" Nov 03 21:09:16.055860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:16.055824 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" path="/var/lib/kubelet/pods/afb3b94d-a194-410a-aecf-9dfbff512cd2/volumes" Nov 03 21:09:16.069836 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:16.069807 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:09:19.141554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:19.141516 5932 generic.go:356] "Generic (PLEG): container finished" podID="486253bb-d0a1-4d28-a087-c132a05238ea" containerID="b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe" exitCode=0 Nov 03 21:09:19.142091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:19.141558 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" event={"ID":"486253bb-d0a1-4d28-a087-c132a05238ea","Type":"ContainerDied","Data":"b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe"} Nov 03 21:09:19.142091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:19.142043 5932 scope.go:122] "RemoveContainer" containerID="b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe" Nov 03 21:09:19.143691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:19.143662 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:19.143792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:19.143731 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:19.143792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:19.143762 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:20.157357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:20.157318 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" event={"ID":"486253bb-d0a1-4d28-a087-c132a05238ea","Type":"ContainerStarted","Data":"3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76"} Nov 03 21:09:20.418398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:20.418312 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:09:20.418578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:20.418531 5932 patch_prober.go:29] interesting pod/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 21:09:20.418656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:20.418574 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 21:09:21.418541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:21.418498 5932 patch_prober.go:29] interesting pod/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 21:09:21.418983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:21.418557 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 21:09:22.418321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:22.418284 5932 patch_prober.go:29] interesting pod/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 21:09:22.418546 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:22.418362 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 21:09:23.418035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:23.417999 5932 patch_prober.go:29] interesting pod/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 21:09:23.418471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:23.418054 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 21:09:24.417906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:24.417869 5932 patch_prober.go:29] interesting pod/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 21:09:24.418091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:24.417922 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 21:09:25.418451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:25.418417 5932 patch_prober.go:29] interesting pod/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" start-of-body= Nov 03 21:09:25.418871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:25.418471 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.235:15021/healthz/ready\": dial tcp 10.132.0.235:15021: connect: connection refused" Nov 03 21:09:26.423180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:26.423148 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:09:26.423670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:26.423647 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:09:26.424447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:26.424423 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:09:29.858835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:29.858795 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj"] Nov 03 21:09:29.904853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:29.904817 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-9ce672-zs6rj"] Nov 03 21:09:30.053511 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:30.053478 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db702dc3-5fc1-4e2c-bfcb-eb88bf801b47" path="/var/lib/kubelet/pods/db702dc3-5fc1-4e2c-bfcb-eb88bf801b47/volumes" Nov 03 21:09:37.324860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:37.324825 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d"] Nov 03 21:09:37.325366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:37.325046 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" containerID="cri-o://92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9" gracePeriod=30 Nov 03 21:09:38.300345 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.300306 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg"] Nov 03 21:09:38.301064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.301047 5932 cpu_manager.go:414] "containerMap: removing container" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" Nov 03 21:09:38.301120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.301066 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" Nov 03 21:09:38.301176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.301166 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="051ea739-77c3-43ae-bcda-7d1915deeb20" containerName="istio-proxy" Nov 03 21:09:38.301218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.301184 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb3b94d-a194-410a-aecf-9dfbff512cd2" containerName="main" Nov 03 21:09:38.317171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.317142 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.359868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.359824 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg"] Nov 03 21:09:38.366334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.366295 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:09:38.367670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.367635 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdp6q\" (UniqueName: \"kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.367987 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.367965 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.368328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.368304 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.400391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.400356 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:09:38.470099 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.470059 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.470294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.470183 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.470294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.470240 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdp6q\" (UniqueName: \"kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.470930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.470902 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.472792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.472768 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.528092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.528062 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdp6q\" (UniqueName: \"kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q\") pod \"maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:38.632347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:38.632307 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:39.049559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:39.049475 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg"] Nov 03 21:09:39.054713 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:09:39.054683 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76700bcc_5326_46f2_bd40_2b82bb0ee170.slice/crio-3ce0d1e2a098b665d652a48dd7449d3cfcc54b644804f0310abec14cc3aca817 WatchSource:0}: Error finding container 3ce0d1e2a098b665d652a48dd7449d3cfcc54b644804f0310abec14cc3aca817: Status 404 returned error can't find the container with id 3ce0d1e2a098b665d652a48dd7449d3cfcc54b644804f0310abec14cc3aca817 Nov 03 21:09:39.340019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:39.339984 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" event={"ID":"76700bcc-5326-46f2-bd40-2b82bb0ee170","Type":"ContainerStarted","Data":"3ce0d1e2a098b665d652a48dd7449d3cfcc54b644804f0310abec14cc3aca817"} Nov 03 21:09:39.453493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:39.453428 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" podStartSLOduration=1.453409841 podStartE2EDuration="1.453409841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:39.447554486 +0000 UTC m=+2387.512795960" watchObservedRunningTime="2026-11-03 21:09:39.453409841 +0000 UTC m=+2387.518651315" Nov 03 21:09:40.349325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:40.349240 5932 generic.go:356] "Generic (PLEG): container finished" podID="76700bcc-5326-46f2-bd40-2b82bb0ee170" containerID="9ecd1edc12b82bf53ff74eb7a622efa94687cbfd9b7e8672f6039927392191b6" exitCode=0 Nov 03 21:09:40.349475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:40.349379 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" event={"ID":"76700bcc-5326-46f2-bd40-2b82bb0ee170","Type":"ContainerDied","Data":"9ecd1edc12b82bf53ff74eb7a622efa94687cbfd9b7e8672f6039927392191b6"} Nov 03 21:09:41.523654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.523596 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:41.611355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.611264 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q\" (UniqueName: \"kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q\") pod \"76700bcc-5326-46f2-bd40-2b82bb0ee170\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " Nov 03 21:09:41.611355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.611345 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca\") pod \"76700bcc-5326-46f2-bd40-2b82bb0ee170\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " Nov 03 21:09:41.611670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.611534 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca\") pod \"76700bcc-5326-46f2-bd40-2b82bb0ee170\" (UID: \"76700bcc-5326-46f2-bd40-2b82bb0ee170\") " Nov 03 21:09:41.611868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.611829 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca" pod "76700bcc-5326-46f2-bd40-2b82bb0ee170" (UID: "76700bcc-5326-46f2-bd40-2b82bb0ee170"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:41.612244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.612218 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:41.613816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.613789 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca" pod "76700bcc-5326-46f2-bd40-2b82bb0ee170" (UID: "76700bcc-5326-46f2-bd40-2b82bb0ee170"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:41.613902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.613837 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q" pod "76700bcc-5326-46f2-bd40-2b82bb0ee170" (UID: "76700bcc-5326-46f2-bd40-2b82bb0ee170"). InnerVolumeSpecName "kube-api-access-rdp6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:41.713484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.713441 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76700bcc-5326-46f2-bd40-2b82bb0ee170-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:41.713484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.713479 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdp6q\" (UniqueName: \"kubernetes.io/projected/76700bcc-5326-46f2-bd40-2b82bb0ee170-kube-api-access-rdp6q\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:41.762617 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.762573 5932 patch_prober.go:29] interesting pod/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:09:41.762855 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:41.762659 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:09:42.367992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.367963 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" Nov 03 21:09:42.368167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.367963 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg" event={"ID":"76700bcc-5326-46f2-bd40-2b82bb0ee170","Type":"ContainerDied","Data":"3ce0d1e2a098b665d652a48dd7449d3cfcc54b644804f0310abec14cc3aca817"} Nov 03 21:09:42.368167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.368082 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce0d1e2a098b665d652a48dd7449d3cfcc54b644804f0310abec14cc3aca817" Nov 03 21:09:42.628812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.628783 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 21:09:42.726854 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.726815 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.726899 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.726965 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727000 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727040 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727147 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:42.727295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727189 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727281 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:42.727417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727300 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727348 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:42.727417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727375 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727420 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy\") pod \"5a834d62-437d-4854-a49d-a23856e72225\" (UID: \"5a834d62-437d-4854-a49d-a23856e72225\") " Nov 03 21:09:42.727690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727513 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:42.727857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.727832 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:42.728116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.728094 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.728224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.728120 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.728224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.728135 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.728224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.728148 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.728224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.728161 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a834d62-437d-4854-a49d-a23856e72225-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.729481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.729457 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:42.729751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.729730 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:42.729846 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.729763 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "kube-api-access-nvmfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:42.729945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.729923 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo" pod "5a834d62-437d-4854-a49d-a23856e72225" (UID: "5a834d62-437d-4854-a49d-a23856e72225"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:09:42.793025 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.792992 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8"] Nov 03 21:09:42.796671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.796637 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb"] Nov 03 21:09:42.829031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.828993 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a834d62-437d-4854-a49d-a23856e72225-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.829031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.829024 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.829031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.829034 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvmfw\" (UniqueName: \"kubernetes.io/projected/5a834d62-437d-4854-a49d-a23856e72225-kube-api-access-nvmfw\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:42.829031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:42.829043 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a834d62-437d-4854-a49d-a23856e72225-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:43.085319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.085279 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm"] Nov 03 21:09:43.085992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.085969 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" Nov 03 21:09:43.085992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.085985 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" Nov 03 21:09:43.086197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.086007 5932 cpu_manager.go:414] "containerMap: removing container" podUID="76700bcc-5326-46f2-bd40-2b82bb0ee170" containerName="revoke-keys" Nov 03 21:09:43.086197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.086013 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76700bcc-5326-46f2-bd40-2b82bb0ee170" containerName="revoke-keys" Nov 03 21:09:43.086197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.086133 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" Nov 03 21:09:43.086197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.086142 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" Nov 03 21:09:43.086197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.086152 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="76700bcc-5326-46f2-bd40-2b82bb0ee170" containerName="revoke-keys" Nov 03 21:09:43.089017 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.088992 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.271672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.266564 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.271672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.266746 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f1855-openshift-default-dockercfg-n94fj\"" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.286778 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.286896 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvrj\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.286994 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.287055 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.287130 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.287161 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.287256 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.287654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.287307 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.315985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.312017 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm"] Nov 03 21:09:43.389693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389659 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.389886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389744 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvrj\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.389886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389804 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.389886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389843 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389895 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389918 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.389990 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390034 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390121 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390175 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390213 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390435 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.390535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390512 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.391002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.390982 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.392894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.392870 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.393005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.392955 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.398190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.398159 5932 generic.go:356] "Generic (PLEG): container finished" podID="5a834d62-437d-4854-a49d-a23856e72225" containerID="92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9" exitCode=0 Nov 03 21:09:43.398335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.398232 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" Nov 03 21:09:43.398335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.398298 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" event={"ID":"5a834d62-437d-4854-a49d-a23856e72225","Type":"ContainerDied","Data":"92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9"} Nov 03 21:09:43.398438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.398382 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d" event={"ID":"5a834d62-437d-4854-a49d-a23856e72225","Type":"ContainerDied","Data":"2aef70dad606054dd660f21de5587e70a7a8856dfafd97b3c8b75f91d7e1da87"} Nov 03 21:09:43.398438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.398409 5932 scope.go:122] "RemoveContainer" containerID="92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9" Nov 03 21:09:43.415938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.415908 5932 scope.go:122] "RemoveContainer" containerID="d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d" Nov 03 21:09:43.426089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.426068 5932 scope.go:122] "RemoveContainer" containerID="92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9" Nov 03 21:09:43.426428 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:09:43.426406 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9\": container with ID starting with 92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9 not found: ID does not exist" containerID="92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9" Nov 03 21:09:43.426511 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.426442 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9"} err="failed to get container status \"92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9\": rpc error: code = NotFound desc = could not find container \"92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9\": container with ID starting with 92b1f5df7aabe0d6e28d528329e002bf4f9bcb130b7adc74524f80e991d089d9 not found: ID does not exist" Nov 03 21:09:43.426511 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.426469 5932 scope.go:122] "RemoveContainer" containerID="d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d" Nov 03 21:09:43.426769 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:09:43.426747 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d\": container with ID starting with d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d not found: ID does not exist" containerID="d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d" Nov 03 21:09:43.426819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.426775 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d"} err="failed to get container status \"d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d\": rpc error: code = NotFound desc = could not find container \"d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d\": container with ID starting with d1213efa639bafea2d30accad9dffaa1e8c3a2adafddb31386c84eb4a5f9d11d not found: ID does not exist" Nov 03 21:09:43.447899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.447859 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.448048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.447964 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvrj\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:43.554829 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.554786 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d"] Nov 03 21:09:43.559737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.559703 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-9ce672-openshift-default-65d6bdf96c-29c9d"] Nov 03 21:09:43.704794 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:43.704704 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:44.053047 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.052965 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a834d62-437d-4854-a49d-a23856e72225" path="/var/lib/kubelet/pods/5a834d62-437d-4854-a49d-a23856e72225/volumes" Nov 03 21:09:44.136497 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.136461 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm"] Nov 03 21:09:44.141662 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:09:44.141615 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d01288_f79b_44ca_8aa6_ebad2ce91abe.slice/crio-068e455a4d7b18f3259fa296ece40e8fcfe3c657bd475d60f94deebd71899e40 WatchSource:0}: Error finding container 068e455a4d7b18f3259fa296ece40e8fcfe3c657bd475d60f94deebd71899e40: Status 404 returned error can't find the container with id 068e455a4d7b18f3259fa296ece40e8fcfe3c657bd475d60f94deebd71899e40 Nov 03 21:09:44.145346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.145304 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:44.145490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.145381 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:44.145490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.145409 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:44.158018 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:09:44.157982 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:09:44.409213 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.409172 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" event={"ID":"19d01288-f79b-44ca-8aa6-ebad2ce91abe","Type":"ContainerStarted","Data":"16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f"} Nov 03 21:09:44.409427 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.409238 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" event={"ID":"19d01288-f79b-44ca-8aa6-ebad2ce91abe","Type":"ContainerStarted","Data":"068e455a4d7b18f3259fa296ece40e8fcfe3c657bd475d60f94deebd71899e40"} Nov 03 21:09:44.527338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.527280 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podStartSLOduration=1.527263539 podStartE2EDuration="1.527263539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:44.523007193 +0000 UTC m=+2392.588248666" watchObservedRunningTime="2026-11-03 21:09:44.527263539 +0000 UTC m=+2392.592505012" Nov 03 21:09:44.704897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:44.704802 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:45.712824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:45.712790 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:46.473136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:46.463727 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:46.473136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:46.464808 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:09:48.849645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:48.848692 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn"] Nov 03 21:09:48.861654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:48.860990 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" Nov 03 21:09:48.861654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:48.861029 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a834d62-437d-4854-a49d-a23856e72225" containerName="istio-proxy" Nov 03 21:09:48.869646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:48.866928 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:48.924775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:48.924738 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-10fa59-openshift-default-dockercfg-jj4x9\"" Nov 03 21:09:49.098646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.063303 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.099147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.099093 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.114697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.105055 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.114697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.105253 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.114697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.105291 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.140649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.121774 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.140649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.121855 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-854cs\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.140649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.121924 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.140649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.122012 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.246648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.223521 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.251822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.251783 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.277843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.277807 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.278260 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.278103 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.278891 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.278870 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.284318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.247356 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.285536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.284904 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.285536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.285269 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.288236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.288218 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.288385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.288370 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.288489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.288479 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-854cs\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.288594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.288584 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.289545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.288326 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.289837 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.289815 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn"] Nov 03 21:09:49.290401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.290382 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.293097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.292382 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.293666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.293646 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.377583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.377538 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-854cs\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.378524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.378502 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token\") pod \"e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:49.500997 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:49.500929 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:50.045648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.044786 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn"] Nov 03 21:09:50.084544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.084497 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:50.084706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.084602 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:50.084706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.084668 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:09:50.275653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.263480 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2"] Nov 03 21:09:50.809285 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.809249 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" event={"ID":"660352ac-d467-4113-bbf4-f8b164da8f70","Type":"ContainerStarted","Data":"4f4d4bb9189513dcd920d6df77ad6fe6727741cc8196f1991fa5261bea76cc83"} Nov 03 21:09:50.809595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:50.809559 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" event={"ID":"660352ac-d467-4113-bbf4-f8b164da8f70","Type":"ContainerStarted","Data":"df687f57878d3a620a3fe4b9c46b08c240b6704cae3b82f1b987d39a2081b4e5"} Nov 03 21:09:51.501652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:51.501603 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:51.507960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:51.507931 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:51.631702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:51.629613 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podStartSLOduration=3.629588862 podStartE2EDuration="3.629588862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:09:50.946987333 +0000 UTC m=+2399.012228806" watchObservedRunningTime="2026-11-03 21:09:51.629588862 +0000 UTC m=+2399.694830337" Nov 03 21:09:51.817423 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:51.817341 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:51.828659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:51.828608 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:09:53.082732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.082688 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9"] Nov 03 21:09:53.088245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.088202 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.093295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.093261 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9"] Nov 03 21:09:53.119999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.118958 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpnj\" (UniqueName: \"kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.119999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.119034 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.119999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.119296 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.155738 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.155703 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:09:53.156063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.156042 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:09:53.220714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.220670 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.221126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.221100 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpnj\" (UniqueName: \"kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.221295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.221280 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.222385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.222358 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.282643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.282576 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.283926 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.283897 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpnj\" (UniqueName: \"kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj\") pod \"maas-api-revoke-sub-0cb5f5e3-hpzj9\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.406982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.406949 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:53.918903 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:53.918334 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9"] Nov 03 21:09:54.911776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:54.911735 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" event={"ID":"1ff25ee5-3611-440d-a49b-0e4758a71849","Type":"ContainerStarted","Data":"57aeea164fa703a8045fe154638067ddb3b2ef8383067409fac9a94c5444b329"} Nov 03 21:09:55.278776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:55.278347 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerName="payload-pre-processing" containerID="cri-o://7cad49f3fc874c19eac9b87dff1cdadc7d56f7fdb416989e04714f8845cdebd1" gracePeriod=25 Nov 03 21:09:55.925112 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:55.925060 5932 generic.go:356] "Generic (PLEG): container finished" podID="1ff25ee5-3611-440d-a49b-0e4758a71849" containerID="a2ba49b66994f08e2ae493025c77fd35153a6e0f3a2c7d2c916aaf17b0aacbd1" exitCode=0 Nov 03 21:09:55.925564 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:55.925130 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" event={"ID":"1ff25ee5-3611-440d-a49b-0e4758a71849","Type":"ContainerDied","Data":"a2ba49b66994f08e2ae493025c77fd35153a6e0f3a2c7d2c916aaf17b0aacbd1"} Nov 03 21:09:56.092821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:56.092787 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:09:56.103129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:56.103099 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:09:57.126817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.126783 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:57.169161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.169109 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca\") pod \"1ff25ee5-3611-440d-a49b-0e4758a71849\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " Nov 03 21:09:57.169379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.169174 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj\" (UniqueName: \"kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj\") pod \"1ff25ee5-3611-440d-a49b-0e4758a71849\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " Nov 03 21:09:57.169458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.169441 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca\") pod \"1ff25ee5-3611-440d-a49b-0e4758a71849\" (UID: \"1ff25ee5-3611-440d-a49b-0e4758a71849\") " Nov 03 21:09:57.170543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.170510 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca" pod "1ff25ee5-3611-440d-a49b-0e4758a71849" (UID: "1ff25ee5-3611-440d-a49b-0e4758a71849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:09:57.172296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.172251 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj" pod "1ff25ee5-3611-440d-a49b-0e4758a71849" (UID: "1ff25ee5-3611-440d-a49b-0e4758a71849"). InnerVolumeSpecName "kube-api-access-zmpnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:57.172438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.172340 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca" pod "1ff25ee5-3611-440d-a49b-0e4758a71849" (UID: "1ff25ee5-3611-440d-a49b-0e4758a71849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:57.271652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.271542 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:57.271652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.271588 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpnj\" (UniqueName: \"kubernetes.io/projected/1ff25ee5-3611-440d-a49b-0e4758a71849-kube-api-access-zmpnj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:57.271652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.271603 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ff25ee5-3611-440d-a49b-0e4758a71849-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:57.818214 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.818166 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" containerID="cri-o://ba3e4672fce56e0ce5e08b18377d0f522e2976d1bce3c7ad53dc224078cce6f5" gracePeriod=45 Nov 03 21:09:57.821801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.821757 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" containerID="cri-o://012ec6e1578a8dd721e79500db25289d0ce7ef630ea929e76a559d246b97a93b" gracePeriod=45 Nov 03 21:09:57.951571 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.951530 5932 generic.go:356] "Generic (PLEG): container finished" podID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerID="ba3e4672fce56e0ce5e08b18377d0f522e2976d1bce3c7ad53dc224078cce6f5" exitCode=0 Nov 03 21:09:57.951820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.951668 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" event={"ID":"2e830b87-2d2a-4d61-9171-82ce54bdba23","Type":"ContainerDied","Data":"ba3e4672fce56e0ce5e08b18377d0f522e2976d1bce3c7ad53dc224078cce6f5"} Nov 03 21:09:57.954068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.954031 5932 generic.go:356] "Generic (PLEG): container finished" podID="02a51e7f-7796-446d-a553-bb712200a85a" containerID="012ec6e1578a8dd721e79500db25289d0ce7ef630ea929e76a559d246b97a93b" exitCode=0 Nov 03 21:09:57.954359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.954333 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" event={"ID":"02a51e7f-7796-446d-a553-bb712200a85a","Type":"ContainerDied","Data":"012ec6e1578a8dd721e79500db25289d0ce7ef630ea929e76a559d246b97a93b"} Nov 03 21:09:57.957385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.957356 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" event={"ID":"1ff25ee5-3611-440d-a49b-0e4758a71849","Type":"ContainerDied","Data":"57aeea164fa703a8045fe154638067ddb3b2ef8383067409fac9a94c5444b329"} Nov 03 21:09:57.957549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.957445 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57aeea164fa703a8045fe154638067ddb3b2ef8383067409fac9a94c5444b329" Nov 03 21:09:57.957549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:57.957462 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9" Nov 03 21:09:58.178451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.178420 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:09:58.184259 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.184226 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 21:09:58.286017 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.285977 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache\") pod \"02a51e7f-7796-446d-a553-bb712200a85a\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " Nov 03 21:09:58.286231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286108 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm\") pod \"2e830b87-2d2a-4d61-9171-82ce54bdba23\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " Nov 03 21:09:58.286231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286152 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb\" (UniqueName: \"kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb\") pod \"2e830b87-2d2a-4d61-9171-82ce54bdba23\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " Nov 03 21:09:58.286231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286211 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir\") pod \"2e830b87-2d2a-4d61-9171-82ce54bdba23\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " Nov 03 21:09:58.286435 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286277 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache" pod "02a51e7f-7796-446d-a553-bb712200a85a" (UID: "02a51e7f-7796-446d-a553-bb712200a85a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.286435 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286292 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs\") pod \"2e830b87-2d2a-4d61-9171-82ce54bdba23\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " Nov 03 21:09:58.286435 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286366 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm\") pod \"02a51e7f-7796-446d-a553-bb712200a85a\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " Nov 03 21:09:58.286700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286432 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache\") pod \"2e830b87-2d2a-4d61-9171-82ce54bdba23\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " Nov 03 21:09:58.286700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286478 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir\") pod \"02a51e7f-7796-446d-a553-bb712200a85a\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " Nov 03 21:09:58.286700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286520 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home\") pod \"02a51e7f-7796-446d-a553-bb712200a85a\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " Nov 03 21:09:58.286700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286602 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs\") pod \"02a51e7f-7796-446d-a553-bb712200a85a\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " Nov 03 21:09:58.286700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286668 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir" pod "2e830b87-2d2a-4d61-9171-82ce54bdba23" (UID: "2e830b87-2d2a-4d61-9171-82ce54bdba23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.286700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286691 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9\" (UniqueName: \"kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9\") pod \"02a51e7f-7796-446d-a553-bb712200a85a\" (UID: \"02a51e7f-7796-446d-a553-bb712200a85a\") " Nov 03 21:09:58.287133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286726 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home\") pod \"2e830b87-2d2a-4d61-9171-82ce54bdba23\" (UID: \"2e830b87-2d2a-4d61-9171-82ce54bdba23\") " Nov 03 21:09:58.287133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286874 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir" pod "02a51e7f-7796-446d-a553-bb712200a85a" (UID: "02a51e7f-7796-446d-a553-bb712200a85a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.287133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.286962 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home" pod "02a51e7f-7796-446d-a553-bb712200a85a" (UID: "02a51e7f-7796-446d-a553-bb712200a85a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.287312 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287156 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home" pod "2e830b87-2d2a-4d61-9171-82ce54bdba23" (UID: "2e830b87-2d2a-4d61-9171-82ce54bdba23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.287312 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287282 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache" pod "2e830b87-2d2a-4d61-9171-82ce54bdba23" (UID: "2e830b87-2d2a-4d61-9171-82ce54bdba23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.287907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287876 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.288033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287909 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.288033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287924 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.288033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287936 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.288033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.287949 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.289313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.289274 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm" pod "02a51e7f-7796-446d-a553-bb712200a85a" (UID: "02a51e7f-7796-446d-a553-bb712200a85a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.289684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.289643 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs" pod "02a51e7f-7796-446d-a553-bb712200a85a" (UID: "02a51e7f-7796-446d-a553-bb712200a85a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:58.289684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.289667 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm" pod "2e830b87-2d2a-4d61-9171-82ce54bdba23" (UID: "2e830b87-2d2a-4d61-9171-82ce54bdba23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:09:58.290006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.289978 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb" pod "2e830b87-2d2a-4d61-9171-82ce54bdba23" (UID: "2e830b87-2d2a-4d61-9171-82ce54bdba23"). InnerVolumeSpecName "kube-api-access-mjclb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:58.290238 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.290221 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9" pod "02a51e7f-7796-446d-a553-bb712200a85a" (UID: "02a51e7f-7796-446d-a553-bb712200a85a"). InnerVolumeSpecName "kube-api-access-wz8j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:09:58.291379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.291352 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs" pod "2e830b87-2d2a-4d61-9171-82ce54bdba23" (UID: "2e830b87-2d2a-4d61-9171-82ce54bdba23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:09:58.389656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389580 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjclb\" (UniqueName: \"kubernetes.io/projected/2e830b87-2d2a-4d61-9171-82ce54bdba23-kube-api-access-mjclb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.389656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389645 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.389656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389661 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e830b87-2d2a-4d61-9171-82ce54bdba23-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.389989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389674 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a51e7f-7796-446d-a553-bb712200a85a-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.389989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389687 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a51e7f-7796-446d-a553-bb712200a85a-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.389989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389700 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz8j9\" (UniqueName: \"kubernetes.io/projected/02a51e7f-7796-446d-a553-bb712200a85a-kube-api-access-wz8j9\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.389989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.389715 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e830b87-2d2a-4d61-9171-82ce54bdba23-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:09:58.467118 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.467078 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h"] Nov 03 21:09:58.478493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.478456 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj"] Nov 03 21:09:58.478978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.478943 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" containerID="cri-o://dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71" gracePeriod=30 Nov 03 21:09:58.485825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.485400 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs"] Nov 03 21:09:58.485825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.485785 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" containerID="cri-o://96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a" gracePeriod=30 Nov 03 21:09:58.969887 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.969858 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" Nov 03 21:09:58.970075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.969932 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8" event={"ID":"02a51e7f-7796-446d-a553-bb712200a85a","Type":"ContainerDied","Data":"bac9e5cdaa7c0be3f425a3c88588ff4edab950f6429f619b50154b2e403eeb04"} Nov 03 21:09:58.970075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.969995 5932 scope.go:122] "RemoveContainer" containerID="012ec6e1578a8dd721e79500db25289d0ce7ef630ea929e76a559d246b97a93b" Nov 03 21:09:58.975587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.972324 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" Nov 03 21:09:58.975587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.972373 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" event={"ID":"2e830b87-2d2a-4d61-9171-82ce54bdba23","Type":"ContainerDied","Data":"c8ad4be10566ba0d57d573be43d007ead2781539387ecb6d7a9cd7726b64e54b"} Nov 03 21:09:58.975587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.972435 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" containerID="cri-o://a4e1e79577e1ee7fe5bea83e9427c59a6acbab6d246b650d1e15d99a182d890a" gracePeriod=30 Nov 03 21:09:58.986699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:58.986670 5932 scope.go:122] "RemoveContainer" containerID="ba3e4672fce56e0ce5e08b18377d0f522e2976d1bce3c7ad53dc224078cce6f5" Nov 03 21:09:59.163920 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.163879 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded" Nov 03 21:09:59.249696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.248809 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8"] Nov 03 21:09:59.255741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.255701 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/distinct-2-w5-2a5ec9-kserve-75c8b769c6-cpwd8"] Nov 03 21:09:59.374256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.374221 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb"] Nov 03 21:09:59.380668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.380615 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-2a5ec9/simulator-w5-2a5ec9-kserve-6f6b8d94f-sllmb"] Nov 03 21:09:59.732912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.732875 5932 patch_prober.go:29] interesting pod/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:09:59.733126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.732947 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:09:59.767659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.767612 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm"] Nov 03 21:09:59.768453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768428 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1ff25ee5-3611-440d-a49b-0e4758a71849" containerName="revoke-keys" Nov 03 21:09:59.768453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768451 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ff25ee5-3611-440d-a49b-0e4758a71849" containerName="revoke-keys" Nov 03 21:09:59.768606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768494 5932 cpu_manager.go:414] "containerMap: removing container" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" Nov 03 21:09:59.768606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768499 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" Nov 03 21:09:59.768606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768512 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" Nov 03 21:09:59.768606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768517 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" Nov 03 21:09:59.768780 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768618 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a51e7f-7796-446d-a553-bb712200a85a" containerName="main" Nov 03 21:09:59.768780 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768647 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" containerName="main" Nov 03 21:09:59.768780 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.768663 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ff25ee5-3611-440d-a49b-0e4758a71849" containerName="revoke-keys" Nov 03 21:09:59.771245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.771228 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:09:59.823222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.823181 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-10fa59\"" Nov 03 21:09:59.826327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.826294 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm"] Nov 03 21:09:59.838748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.838713 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dp28\" (UniqueName: \"kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28\") pod \"payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:09:59.838904 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.838768 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:09:59.850542 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.850471 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-10fa59-dockercfg-b69fx\"" Nov 03 21:09:59.951648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.948565 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dp28\" (UniqueName: \"kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28\") pod \"payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:09:59.951648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.948640 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:09:59.951648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.949914 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:09:59.964342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.957934 5932 scope.go:122] "RemoveContainer" containerID="4c5a02fd62c07a95e3071427e552f854648326643dbe00a797b46bf33a589f1a" Nov 03 21:09:59.976928 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:09:59.964857 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk"] Nov 03 21:10:00.015461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.015421 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dp28\" (UniqueName: \"kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28\") pod \"payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:10:00.024524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.024487 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk"] Nov 03 21:10:00.025515 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.025486 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.038489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.038457 5932 scope.go:122] "RemoveContainer" containerID="4657a01f6cfba84146e497a84b1ec111215d3772984b429659eeb89c6d769c60" Nov 03 21:10:00.041018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.040991 5932 generic.go:356] "Generic (PLEG): container finished" podID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerID="5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401" exitCode=0 Nov 03 21:10:00.041153 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.041059 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" event={"ID":"d7ab015e-f0c9-4936-8734-d1964c9cd4f1","Type":"ContainerDied","Data":"5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401"} Nov 03 21:10:00.041510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.041490 5932 scope.go:122] "RemoveContainer" containerID="5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401" Nov 03 21:10:00.044155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.044123 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:00.044261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.044215 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:00.044327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.044269 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:00.049672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.049608 5932 generic.go:356] "Generic (PLEG): container finished" podID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerID="46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee" exitCode=0 Nov 03 21:10:00.052204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.052186 5932 scope.go:122] "RemoveContainer" containerID="2d8bddc041c9f5e84b9aeb50751b20684e46a032e396aa24fdf593326f910c8f" Nov 03 21:10:00.055022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.054926 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a51e7f-7796-446d-a553-bb712200a85a" path="/var/lib/kubelet/pods/02a51e7f-7796-446d-a553-bb712200a85a/volumes" Nov 03 21:10:00.055496 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.055475 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e830b87-2d2a-4d61-9171-82ce54bdba23" path="/var/lib/kubelet/pods/2e830b87-2d2a-4d61-9171-82ce54bdba23/volumes" Nov 03 21:10:00.056007 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.055978 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" event={"ID":"d0e6377a-7d4b-4dde-9d95-9c3b271a095b","Type":"ContainerDied","Data":"46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee"} Nov 03 21:10:00.056736 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.056451 5932 scope.go:122] "RemoveContainer" containerID="46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee" Nov 03 21:10:00.060398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.060220 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:00.060398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.060317 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:00.060398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.060365 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:00.066430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.066404 5932 scope.go:122] "RemoveContainer" containerID="3851d56ebeff1f2184eaec8ab3e3c64e9b1f016db85a2608157864ed4c0ce4b8" Nov 03 21:10:00.149264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.149225 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:10:00.153715 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.151669 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc6m8\" (UniqueName: \"kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8\") pod \"payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.153715 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.151731 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.255682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.253449 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc6m8\" (UniqueName: \"kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8\") pod \"payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.255682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.253526 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.255682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.254570 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.333530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.328864 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc6m8\" (UniqueName: \"kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8\") pod \"payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.356548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.356440 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:00.587775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.587714 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm"] Nov 03 21:10:00.590903 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:10:00.590873 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc207bcd6_8186_4416_b183_d806d5ff6f28.slice/crio-4a6e8a384bd64a4969db9bb7ea7e8bdb86c78f1345526a9f56e2ddecc8d44c25 WatchSource:0}: Error finding container 4a6e8a384bd64a4969db9bb7ea7e8bdb86c78f1345526a9f56e2ddecc8d44c25: Status 404 returned error can't find the container with id 4a6e8a384bd64a4969db9bb7ea7e8bdb86c78f1345526a9f56e2ddecc8d44c25 Nov 03 21:10:00.804649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:00.804593 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk"] Nov 03 21:10:00.810327 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:10:00.810280 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844b4c14_d200_405c_816b_9297609dbe3d.slice/crio-1df517f4d44f39c1821f3244a84df5ef28aedd2555985293336eb2de9547479b WatchSource:0}: Error finding container 1df517f4d44f39c1821f3244a84df5ef28aedd2555985293336eb2de9547479b: Status 404 returned error can't find the container with id 1df517f4d44f39c1821f3244a84df5ef28aedd2555985293336eb2de9547479b Nov 03 21:10:01.085954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.085918 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" event={"ID":"844b4c14-d200-405c-816b-9297609dbe3d","Type":"ContainerStarted","Data":"1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01"} Nov 03 21:10:01.086158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.085977 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" event={"ID":"844b4c14-d200-405c-816b-9297609dbe3d","Type":"ContainerStarted","Data":"1df517f4d44f39c1821f3244a84df5ef28aedd2555985293336eb2de9547479b"} Nov 03 21:10:01.086158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.086080 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:01.087918 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.087891 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" event={"ID":"c207bcd6-8186-4416-b183-d806d5ff6f28","Type":"ContainerStarted","Data":"d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3"} Nov 03 21:10:01.088038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.087934 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" event={"ID":"c207bcd6-8186-4416-b183-d806d5ff6f28","Type":"ContainerStarted","Data":"4a6e8a384bd64a4969db9bb7ea7e8bdb86c78f1345526a9f56e2ddecc8d44c25"} Nov 03 21:10:01.088503 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.088487 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:10:01.090402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.090376 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" event={"ID":"d7ab015e-f0c9-4936-8734-d1964c9cd4f1","Type":"ContainerStarted","Data":"810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02"} Nov 03 21:10:01.092269 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.092248 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" event={"ID":"d0e6377a-7d4b-4dde-9d95-9c3b271a095b","Type":"ContainerStarted","Data":"708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588"} Nov 03 21:10:01.207098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.206981 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" podStartSLOduration=2.206962751 podStartE2EDuration="2.206962751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:01.204370969 +0000 UTC m=+2409.269612439" watchObservedRunningTime="2026-11-03 21:10:01.206962751 +0000 UTC m=+2409.272204228" Nov 03 21:10:01.299038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.299000 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:10:01.300593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.300563 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 21:10:01.300669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.300635 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 21:10:01.378648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.377921 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:10:01.397020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.395385 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:10:01.397020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.395486 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:10:01.479400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:01.479279 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" podStartSLOduration=2.4792616020000002 podStartE2EDuration="2.479261602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:01.46300166 +0000 UTC m=+2409.528243136" watchObservedRunningTime="2026-11-03 21:10:01.479261602 +0000 UTC m=+2409.544503074" Nov 03 21:10:02.299471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:02.299427 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 21:10:02.299967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:02.299495 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 21:10:02.377933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:02.377890 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:10:02.378096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:02.377974 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:10:03.299381 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:03.299344 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 21:10:03.299576 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:03.299393 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 21:10:03.378157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:03.378111 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:10:03.378344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:03.378177 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:10:04.095737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.094920 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:10:04.102100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.102001 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:10:04.108985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.108956 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.109215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109025 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.109215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109083 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.109215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109159 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.109422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109237 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.109422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109341 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.109422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109345 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:04.109590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109423 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.109590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109434 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.109590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109467 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:04.109590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109499 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.109833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109576 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.109833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109677 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.109833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109732 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.109833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109794 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.109833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109824 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.110095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109884 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.110095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109919 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.110095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.109975 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.110095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.110029 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token\") pod \"41f1a494-1627-4a59-9e2d-2d54db5545c1\" (UID: \"41f1a494-1627-4a59-9e2d-2d54db5545c1\") " Nov 03 21:10:04.110095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.110087 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo\") pod \"a10562d1-1774-4086-90ee-71cf548fd839\" (UID: \"a10562d1-1774-4086-90ee-71cf548fd839\") " Nov 03 21:10:04.111272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.111117 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.111444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.111415 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.111444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.111423 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.112177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.111555 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.112177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.112005 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:04.113146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.112407 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.113146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.112550 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.113146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.112696 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.114497 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.114470 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.114840 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.114815 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "kube-api-access-qgt84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:04.115356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115328 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115363 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115378 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115392 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115410 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115428 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115447 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115463 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115477 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgt84\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-kube-api-access-qgt84\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115491 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41f1a494-1627-4a59-9e2d-2d54db5545c1-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115505 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a10562d1-1774-4086-90ee-71cf548fd839-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115518 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.115710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.115531 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.117723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.117700 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.117723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.117708 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:04.117884 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.117753 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token" pod "a10562d1-1774-4086-90ee-71cf548fd839" (UID: "a10562d1-1774-4086-90ee-71cf548fd839"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:04.117884 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.117769 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "kube-api-access-vndxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:04.118163 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.118138 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token" pod "41f1a494-1627-4a59-9e2d-2d54db5545c1" (UID: "41f1a494-1627-4a59-9e2d-2d54db5545c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:04.194798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.191191 5932 generic.go:356] "Generic (PLEG): container finished" podID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerID="96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a" exitCode=0 Nov 03 21:10:04.194798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.191351 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" event={"ID":"41f1a494-1627-4a59-9e2d-2d54db5545c1","Type":"ContainerDied","Data":"96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a"} Nov 03 21:10:04.194798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.191394 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" event={"ID":"41f1a494-1627-4a59-9e2d-2d54db5545c1","Type":"ContainerDied","Data":"f13b828cd7475959c2792a04f16152bb720edaa47c8523f1dd51244c3f06e937"} Nov 03 21:10:04.194798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.191421 5932 scope.go:122] "RemoveContainer" containerID="96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a" Nov 03 21:10:04.194798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.191581 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs" Nov 03 21:10:04.195388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.195002 5932 generic.go:356] "Generic (PLEG): container finished" podID="a10562d1-1774-4086-90ee-71cf548fd839" containerID="dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71" exitCode=0 Nov 03 21:10:04.195388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.195117 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" event={"ID":"a10562d1-1774-4086-90ee-71cf548fd839","Type":"ContainerDied","Data":"dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71"} Nov 03 21:10:04.195388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.195169 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" event={"ID":"a10562d1-1774-4086-90ee-71cf548fd839","Type":"ContainerDied","Data":"8f1cb96eb4527613ee01bd6e8d682996cb5e361f1f87b2bf15f9f83095f5a454"} Nov 03 21:10:04.195388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.195257 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj" Nov 03 21:10:04.301019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.293656 5932 scope.go:122] "RemoveContainer" containerID="96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a" Nov 03 21:10:04.301019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.294919 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a10562d1-1774-4086-90ee-71cf548fd839-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.301019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.294950 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.301019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.294965 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a10562d1-1774-4086-90ee-71cf548fd839-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.301019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.294979 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vndxd\" (UniqueName: \"kubernetes.io/projected/41f1a494-1627-4a59-9e2d-2d54db5545c1-kube-api-access-vndxd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.301019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.294994 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a10562d1-1774-4086-90ee-71cf548fd839-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.301608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.301308 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 21:10:04.301608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.301359 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 21:10:04.302517 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:10:04.302486 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a\": container with ID starting with 96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a not found: ID does not exist" containerID="96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a" Nov 03 21:10:04.302742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.302718 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a"} err="failed to get container status \"96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a\": rpc error: code = NotFound desc = could not find container \"96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a\": container with ID starting with 96a26f5963b3d97dbe647183283bff21c3f150da11af06d202ac19a12c8e619a not found: ID does not exist" Nov 03 21:10:04.302855 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.302844 5932 scope.go:122] "RemoveContainer" containerID="dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71" Nov 03 21:10:04.305190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.304794 5932 generic.go:356] "Generic (PLEG): container finished" podID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerID="a4e1e79577e1ee7fe5bea83e9427c59a6acbab6d246b650d1e15d99a182d890a" exitCode=0 Nov 03 21:10:04.305190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.304923 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" event={"ID":"03d0cb91-0699-494a-9595-7d06a9c49e60","Type":"ContainerDied","Data":"a4e1e79577e1ee7fe5bea83e9427c59a6acbab6d246b650d1e15d99a182d890a"} Nov 03 21:10:04.386657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.384195 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:10:04.386657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.384254 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:10:04.408541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.408477 5932 scope.go:122] "RemoveContainer" containerID="dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71" Nov 03 21:10:04.412458 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:10:04.412434 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71\": container with ID starting with dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71 not found: ID does not exist" containerID="dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71" Nov 03 21:10:04.412696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.412665 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71"} err="failed to get container status \"dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71\": rpc error: code = NotFound desc = could not find container \"dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71\": container with ID starting with dc7b06fead68ce0f437230e6bae2654f3d5656207a9acea0b5267328e9a73c71 not found: ID does not exist" Nov 03 21:10:04.412830 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.412815 5932 scope.go:122] "RemoveContainer" containerID="86dcd017b8e1a7601353d63cae759d39d27064a7f8f643653073c259b0150647" Nov 03 21:10:04.500865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.500838 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj"] Nov 03 21:10:04.508646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.508502 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-p8dxj"] Nov 03 21:10:04.587550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.587514 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 21:10:04.702088 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702043 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702219 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702291 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702338 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702406 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702469 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702534 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702634 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702591 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.702682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.702663 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy\") pod \"03d0cb91-0699-494a-9595-7d06a9c49e60\" (UID: \"03d0cb91-0699-494a-9595-7d06a9c49e60\") " Nov 03 21:10:04.705394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.704558 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.705394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.704816 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.705394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.704845 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:04.705394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.704882 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.705394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.705042 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.705818 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.705484 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:10:04.712245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.712198 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:10:04.716167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.714768 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "kube-api-access-mwh8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:04.716167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.714787 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs"] Nov 03 21:10:04.717877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.717835 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token" pod "03d0cb91-0699-494a-9595-7d06a9c49e60" (UID: "03d0cb91-0699-494a-9595-7d06a9c49e60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:04.720266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.720242 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-cpwxs"] Nov 03 21:10:04.804942 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.804859 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805117 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwh8s\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-kube-api-access-mwh8s\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805193 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805254 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805308 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805360 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805414 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805478 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805471 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03d0cb91-0699-494a-9595-7d06a9c49e60-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:04.805526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:04.805519 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03d0cb91-0699-494a-9595-7d06a9c49e60-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:05.184979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.184918 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:10:05.299358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.299326 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 21:10:05.299570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.299549 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 21:10:05.317349 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.317319 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" Nov 03 21:10:05.317816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.317325 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h" event={"ID":"03d0cb91-0699-494a-9595-7d06a9c49e60","Type":"ContainerDied","Data":"f730679be0b1542705f0edb00698656d2f622de1daa8fee3b78cbddc154645a9"} Nov 03 21:10:05.317816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.317465 5932 scope.go:122] "RemoveContainer" containerID="a4e1e79577e1ee7fe5bea83e9427c59a6acbab6d246b650d1e15d99a182d890a" Nov 03 21:10:05.384764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.384721 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:10:05.384970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.384778 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:10:05.495954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.495523 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h"] Nov 03 21:10:05.499222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:05.499194 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e68f8112-openshift-default-54c6d8f977-sxx9h"] Nov 03 21:10:06.082568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.082532 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" path="/var/lib/kubelet/pods/03d0cb91-0699-494a-9595-7d06a9c49e60/volumes" Nov 03 21:10:06.083153 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.083135 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" path="/var/lib/kubelet/pods/41f1a494-1627-4a59-9e2d-2d54db5545c1/volumes" Nov 03 21:10:06.083594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.083580 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a10562d1-1774-4086-90ee-71cf548fd839" path="/var/lib/kubelet/pods/a10562d1-1774-4086-90ee-71cf548fd839/volumes" Nov 03 21:10:06.099526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.099488 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:10:06.298934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.298893 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 21:10:06.299157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.298956 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 21:10:06.378278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.378232 5932 patch_prober.go:29] interesting pod/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" start-of-body= Nov 03 21:10:06.378703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:06.378288 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.244:15021/healthz/ready\": dial tcp 10.132.0.244:15021: connect: connection refused" Nov 03 21:10:07.304158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:07.304126 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:10:07.304544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:07.304520 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:10:07.305284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:07.305259 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:10:07.382906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:07.382874 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:10:07.383823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:07.383804 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:10:12.103642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.103595 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:10:12.499207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.499170 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws"] Nov 03 21:10:12.500286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500266 5932 cpu_manager.go:414] "containerMap: removing container" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500289 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500313 5932 cpu_manager.go:414] "containerMap: removing container" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500323 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500347 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500355 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500382 5932 cpu_manager.go:414] "containerMap: removing container" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" Nov 03 21:10:12.500398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500390 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" Nov 03 21:10:12.500724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500548 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="41f1a494-1627-4a59-9e2d-2d54db5545c1" containerName="istio-proxy" Nov 03 21:10:12.500724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500561 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a10562d1-1774-4086-90ee-71cf548fd839" containerName="istio-proxy" Nov 03 21:10:12.500724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500576 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" Nov 03 21:10:12.500724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.500597 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="03d0cb91-0699-494a-9595-7d06a9c49e60" containerName="istio-proxy" Nov 03 21:10:12.506895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.506871 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.550805 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwfl7\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.550883 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.550920 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.550958 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.551016 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.551081 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.551145 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.551193 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.551731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.551259 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.587660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.582703 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws"] Nov 03 21:10:12.670672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.667715 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cfcd54-openshift-default-dockercfg-xg4w6\"" Nov 03 21:10:12.670672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.669575 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.678245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.676808 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfl7\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.678245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.676889 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.687912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.685886 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.688479 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.688425 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.746658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.746608 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfl7\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.753892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.676996 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.754077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.753911 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.754077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.753993 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.754202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.754072 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.754202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.754147 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.754320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.754204 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.754442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.754388 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.755001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.754973 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.755292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.755232 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.757761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.757733 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.758652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.758612 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.811753 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.811687 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token\") pod \"e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:12.823336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:12.822637 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:13.110688 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.110589 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:10:13.257000 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.256968 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws"] Nov 03 21:10:13.259225 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:10:13.259195 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4db7b2e_321b_4631_ad4f_10373b59ae8b.slice/crio-6eb8fa6ca900dd833c4cf37b80fb9acca324d04c1e3fcdc18ab25f44f2b3d469 WatchSource:0}: Error finding container 6eb8fa6ca900dd833c4cf37b80fb9acca324d04c1e3fcdc18ab25f44f2b3d469: Status 404 returned error can't find the container with id 6eb8fa6ca900dd833c4cf37b80fb9acca324d04c1e3fcdc18ab25f44f2b3d469 Nov 03 21:10:13.262714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.262675 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:13.262816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.262770 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:13.262860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.262830 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:13.426438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.426379 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" event={"ID":"c4db7b2e-321b-4631-ad4f-10373b59ae8b","Type":"ContainerStarted","Data":"cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441"} Nov 03 21:10:13.426674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.426464 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" event={"ID":"c4db7b2e-321b-4631-ad4f-10373b59ae8b","Type":"ContainerStarted","Data":"6eb8fa6ca900dd833c4cf37b80fb9acca324d04c1e3fcdc18ab25f44f2b3d469"} Nov 03 21:10:13.541507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.541443 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podStartSLOduration=1.54142328 podStartE2EDuration="1.54142328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:13.53882532 +0000 UTC m=+2421.604066794" watchObservedRunningTime="2026-11-03 21:10:13.54142328 +0000 UTC m=+2421.606664754" Nov 03 21:10:13.823610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:13.823524 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:14.155261 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:10:14.155225 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:10:14.829950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:14.829918 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:15.182880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.182845 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:10:15.445195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.445115 5932 generic.go:356] "Generic (PLEG): container finished" podID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerID="7cad49f3fc874c19eac9b87dff1cdadc7d56f7fdb416989e04714f8845cdebd1" exitCode=0 Nov 03 21:10:15.445342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.445188 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" event={"ID":"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0","Type":"ContainerDied","Data":"7cad49f3fc874c19eac9b87dff1cdadc7d56f7fdb416989e04714f8845cdebd1"} Nov 03 21:10:15.445396 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.445389 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:15.446512 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.446495 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:10:15.564584 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.564551 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 21:10:15.613997 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.613963 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj\" (UniqueName: \"kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj\") pod \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " Nov 03 21:10:15.614187 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.614145 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume\") pod \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\" (UID: \"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0\") " Nov 03 21:10:15.614836 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.614807 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume" pod "310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" (UID: "310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:10:15.616558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.616531 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj" pod "310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" (UID: "310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0"). InnerVolumeSpecName "kube-api-access-4xfqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:10:15.716560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.716474 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xfqj\" (UniqueName: \"kubernetes.io/projected/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-kube-api-access-4xfqj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:15.716560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:15.716508 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:10:16.455638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:16.455596 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" Nov 03 21:10:16.456078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:16.455652 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2" event={"ID":"310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0","Type":"ContainerDied","Data":"3d2218df283ecfdcee8be713b620008e75eb5609ddda4bb45f75c8329f79f2bc"} Nov 03 21:10:16.456078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:16.455728 5932 scope.go:122] "RemoveContainer" containerID="7cad49f3fc874c19eac9b87dff1cdadc7d56f7fdb416989e04714f8845cdebd1" Nov 03 21:10:16.644277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:16.644243 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2"] Nov 03 21:10:16.666952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:16.666910 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-2a5ec9-544c5fb479-7mmq2"] Nov 03 21:10:18.053701 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:18.053670 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" path="/var/lib/kubelet/pods/310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0/volumes" Nov 03 21:10:35.665552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.665469 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb"] Nov 03 21:10:35.666193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.666177 5932 cpu_manager.go:414] "containerMap: removing container" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerName="payload-pre-processing" Nov 03 21:10:35.666249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.666195 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerName="payload-pre-processing" Nov 03 21:10:35.666313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.666304 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="310b02f1-14ef-4dd5-a2e6-fc2b54c0e0f0" containerName="payload-pre-processing" Nov 03 21:10:35.671400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.671382 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.720297 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.720256 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-2a5ec9-openshift-default-dockercfg-kln2c\"" Nov 03 21:10:35.771441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.771408 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb"] Nov 03 21:10:35.803935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.803900 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.803935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.803937 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804188 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.803957 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6stf\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804188 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.804119 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.804216 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.804235 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.804250 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.804380 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.804483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.804439 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.905693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905659 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.905862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905722 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.905862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905802 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.905862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905824 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.905862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905839 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6stf\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905890 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.905991 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906020 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906045 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906211 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906395 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906403 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906457 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.906675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.906618 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.908225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.908207 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.908412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.908385 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.963300 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.963233 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6stf\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.963459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.963441 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token\") pod \"e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:35.985797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:35.985773 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:36.423493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:36.423451 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb"] Nov 03 21:10:36.428543 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:10:36.428509 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fb1ab5_b2bb_421a_add1_07c99039feab.slice/crio-e9400cda2b8b5ac89fc4b4d8d2a4a6d917a3f3b60f0d0c2f9e0c04ab283fb966 WatchSource:0}: Error finding container e9400cda2b8b5ac89fc4b4d8d2a4a6d917a3f3b60f0d0c2f9e0c04ab283fb966: Status 404 returned error can't find the container with id e9400cda2b8b5ac89fc4b4d8d2a4a6d917a3f3b60f0d0c2f9e0c04ab283fb966 Nov 03 21:10:36.431380 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:36.431336 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:36.431468 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:36.431438 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:36.431514 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:36.431482 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:10:37.054679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:37.054637 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" event={"ID":"d1fb1ab5-b2bb-421a-add1-07c99039feab","Type":"ContainerStarted","Data":"c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc"} Nov 03 21:10:37.055079 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:37.054698 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" event={"ID":"d1fb1ab5-b2bb-421a-add1-07c99039feab","Type":"ContainerStarted","Data":"e9400cda2b8b5ac89fc4b4d8d2a4a6d917a3f3b60f0d0c2f9e0c04ab283fb966"} Nov 03 21:10:37.173364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:37.173298 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podStartSLOduration=2.173267869 podStartE2EDuration="2.173267869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:37.167189224 +0000 UTC m=+2445.232430698" watchObservedRunningTime="2026-11-03 21:10:37.173267869 +0000 UTC m=+2445.238509346" Nov 03 21:10:37.986879 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:37.986845 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:37.988935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:37.988908 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 21:10:37.989064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:37.988965 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 21:10:38.986995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:38.986955 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 21:10:38.987481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:38.987027 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 21:10:39.986453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:39.986402 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 21:10:39.986663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:39.986469 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 21:10:40.987066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:40.987030 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 21:10:40.987463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:40.987100 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 21:10:41.986237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:41.986202 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 21:10:41.986414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:41.986269 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 21:10:42.988647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:42.986127 5932 patch_prober.go:29] interesting pod/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Nov 03 21:10:42.988647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:42.986193 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Nov 03 21:10:43.990744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:43.990711 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:43.991192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:43.991175 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:43.991928 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:43.991905 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:10:44.152717 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:10:44.152673 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:10:56.915875 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:56.915839 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn"] Nov 03 21:10:56.920297 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:56.920281 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:56.973880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:56.973847 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn"] Nov 03 21:10:57.015264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.015231 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:10:57.015464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.015271 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:10:57.025393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.025367 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.025514 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.025412 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.025597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.025574 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmslb\" (UniqueName: \"kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.126646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.126587 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.126833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.126670 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.126833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.126731 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmslb\" (UniqueName: \"kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.127353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.127331 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.129117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.129094 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.185870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.185804 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmslb\" (UniqueName: \"kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb\") pod \"maas-api-revoke-sub-9044df0d-5n9qn\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.234425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.234399 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:10:57.622446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:57.622369 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn"] Nov 03 21:10:57.626497 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:10:57.626471 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8288226b_394d_450a_834e_e68b5f402aab.slice/crio-186771b8fa0c486bb74a0c74b17386e634b90c50334f1141f47fbb9953843e00 WatchSource:0}: Error finding container 186771b8fa0c486bb74a0c74b17386e634b90c50334f1141f47fbb9953843e00: Status 404 returned error can't find the container with id 186771b8fa0c486bb74a0c74b17386e634b90c50334f1141f47fbb9953843e00 Nov 03 21:10:58.372358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:58.372278 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" event={"ID":"8288226b-394d-450a-834e-e68b5f402aab","Type":"ContainerStarted","Data":"81e5a78d5f35ae74b4f4ceb12be6a765905d68a7deee362b168ee18b4393046f"} Nov 03 21:10:58.372358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:58.372330 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" event={"ID":"8288226b-394d-450a-834e-e68b5f402aab","Type":"ContainerStarted","Data":"186771b8fa0c486bb74a0c74b17386e634b90c50334f1141f47fbb9953843e00"} Nov 03 21:10:58.483090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:58.483036 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" podStartSLOduration=2.483022634 podStartE2EDuration="2.483022634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:10:58.48035129 +0000 UTC m=+2466.545592762" watchObservedRunningTime="2026-11-03 21:10:58.483022634 +0000 UTC m=+2466.548264145" Nov 03 21:10:59.382458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:59.382427 5932 generic.go:356] "Generic (PLEG): container finished" podID="8288226b-394d-450a-834e-e68b5f402aab" containerID="81e5a78d5f35ae74b4f4ceb12be6a765905d68a7deee362b168ee18b4393046f" exitCode=0 Nov 03 21:10:59.382941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:59.382514 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" event={"ID":"8288226b-394d-450a-834e-e68b5f402aab","Type":"ContainerDied","Data":"81e5a78d5f35ae74b4f4ceb12be6a765905d68a7deee362b168ee18b4393046f"} Nov 03 21:10:59.813378 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:59.813290 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg"] Nov 03 21:10:59.820203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:10:59.820173 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-2a5ec9-dkwpg"] Nov 03 21:11:00.053574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.053540 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76700bcc-5326-46f2-bd40-2b82bb0ee170" path="/var/lib/kubelet/pods/76700bcc-5326-46f2-bd40-2b82bb0ee170/volumes" Nov 03 21:11:00.546273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.546248 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:11:00.704448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.704413 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb\" (UniqueName: \"kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb\") pod \"8288226b-394d-450a-834e-e68b5f402aab\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " Nov 03 21:11:00.704635 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.704552 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca\") pod \"8288226b-394d-450a-834e-e68b5f402aab\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " Nov 03 21:11:00.704635 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.704591 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca\") pod \"8288226b-394d-450a-834e-e68b5f402aab\" (UID: \"8288226b-394d-450a-834e-e68b5f402aab\") " Nov 03 21:11:00.704969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.704936 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca" pod "8288226b-394d-450a-834e-e68b5f402aab" (UID: "8288226b-394d-450a-834e-e68b5f402aab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:00.705375 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.705346 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8288226b-394d-450a-834e-e68b5f402aab-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:00.707045 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.707009 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb" pod "8288226b-394d-450a-834e-e68b5f402aab" (UID: "8288226b-394d-450a-834e-e68b5f402aab"). InnerVolumeSpecName "kube-api-access-mmslb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:00.707241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.707213 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca" pod "8288226b-394d-450a-834e-e68b5f402aab" (UID: "8288226b-394d-450a-834e-e68b5f402aab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:00.806652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.806547 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8288226b-394d-450a-834e-e68b5f402aab-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:00.806652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:00.806579 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmslb\" (UniqueName: \"kubernetes.io/projected/8288226b-394d-450a-834e-e68b5f402aab-kube-api-access-mmslb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:01.400206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:01.400176 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" event={"ID":"8288226b-394d-450a-834e-e68b5f402aab","Type":"ContainerDied","Data":"186771b8fa0c486bb74a0c74b17386e634b90c50334f1141f47fbb9953843e00"} Nov 03 21:11:01.400206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:01.400209 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn" Nov 03 21:11:01.400437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:01.400222 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="186771b8fa0c486bb74a0c74b17386e634b90c50334f1141f47fbb9953843e00" Nov 03 21:11:06.788461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:06.788429 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb"] Nov 03 21:11:06.788998 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:06.788683 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" containerID="cri-o://c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc" gracePeriod=30 Nov 03 21:11:08.178410 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.178380 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc"] Nov 03 21:11:08.179072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.179057 5932 cpu_manager.go:414] "containerMap: removing container" podUID="8288226b-394d-450a-834e-e68b5f402aab" containerName="revoke-keys" Nov 03 21:11:08.179119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.179074 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8288226b-394d-450a-834e-e68b5f402aab" containerName="revoke-keys" Nov 03 21:11:08.179208 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.179200 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="8288226b-394d-450a-834e-e68b5f402aab" containerName="revoke-keys" Nov 03 21:11:08.183948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.183931 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.234446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.234412 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.234905 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.234878 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrq6\" (UniqueName: \"kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.235092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.235073 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.237275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.237249 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc"] Nov 03 21:11:08.278111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.278092 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:11:08.278225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.278155 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:11:08.336711 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.336674 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrq6\" (UniqueName: \"kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.336815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.336744 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.336815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.336763 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.337575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.337550 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.339192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.339169 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.401457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.401433 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrq6\" (UniqueName: \"kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6\") pod \"maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.498006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.497948 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:08.871825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:08.871749 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc"] Nov 03 21:11:08.876671 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:11:08.876639 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45f2b2a_be2f_45bb_bcca_1946e91054dd.slice/crio-b97ceea6fda119c587e97406e8a17098bb17fb864cc54792c2e500022e62f4ba WatchSource:0}: Error finding container b97ceea6fda119c587e97406e8a17098bb17fb864cc54792c2e500022e62f4ba: Status 404 returned error can't find the container with id b97ceea6fda119c587e97406e8a17098bb17fb864cc54792c2e500022e62f4ba Nov 03 21:11:09.471739 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:09.471643 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" event={"ID":"e45f2b2a-be2f-45bb-bcca-1946e91054dd","Type":"ContainerStarted","Data":"a5ab54e85c234628035acf5c320873eaa9fdfc132e1551151d11fc8f45027436"} Nov 03 21:11:09.471739 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:09.471711 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" event={"ID":"e45f2b2a-be2f-45bb-bcca-1946e91054dd","Type":"ContainerStarted","Data":"b97ceea6fda119c587e97406e8a17098bb17fb864cc54792c2e500022e62f4ba"} Nov 03 21:11:09.615164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:09.615089 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" podStartSLOduration=1.615066883 podStartE2EDuration="1.615066883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:09.609372612 +0000 UTC m=+2477.674614085" watchObservedRunningTime="2026-11-03 21:11:09.615066883 +0000 UTC m=+2477.680308356" Nov 03 21:11:10.570208 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:10.570169 5932 generic.go:356] "Generic (PLEG): container finished" podID="e45f2b2a-be2f-45bb-bcca-1946e91054dd" containerID="a5ab54e85c234628035acf5c320873eaa9fdfc132e1551151d11fc8f45027436" exitCode=0 Nov 03 21:11:10.570815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:10.570462 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" event={"ID":"e45f2b2a-be2f-45bb-bcca-1946e91054dd","Type":"ContainerDied","Data":"a5ab54e85c234628035acf5c320873eaa9fdfc132e1551151d11fc8f45027436"} Nov 03 21:11:11.794969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.794943 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:11.890892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.890863 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6\" (UniqueName: \"kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6\") pod \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " Nov 03 21:11:11.891068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.890966 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca\") pod \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " Nov 03 21:11:11.891068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.891016 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca\") pod \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\" (UID: \"e45f2b2a-be2f-45bb-bcca-1946e91054dd\") " Nov 03 21:11:11.891338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.891311 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca" pod "e45f2b2a-be2f-45bb-bcca-1946e91054dd" (UID: "e45f2b2a-be2f-45bb-bcca-1946e91054dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:11.891493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.891479 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:11.893400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.893369 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6" pod "e45f2b2a-be2f-45bb-bcca-1946e91054dd" (UID: "e45f2b2a-be2f-45bb-bcca-1946e91054dd"). InnerVolumeSpecName "kube-api-access-6xrq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:11.893400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.893383 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca" pod "e45f2b2a-be2f-45bb-bcca-1946e91054dd" (UID: "e45f2b2a-be2f-45bb-bcca-1946e91054dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:11.993069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.993033 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e45f2b2a-be2f-45bb-bcca-1946e91054dd-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:11.993069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:11.993066 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xrq6\" (UniqueName: \"kubernetes.io/projected/e45f2b2a-be2f-45bb-bcca-1946e91054dd-kube-api-access-6xrq6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.150690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.147815 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:11:12.195775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.195740 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.195970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.195790 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.195970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.195828 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.195970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.195880 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.195970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.195906 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.196195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.195992 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.196195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196044 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.196195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196062 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:12.196195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196114 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.196467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196261 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data\") pod \"d1fb1ab5-b2bb-421a-add1-07c99039feab\" (UID: \"d1fb1ab5-b2bb-421a-add1-07c99039feab\") " Nov 03 21:11:12.196467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196381 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:12.196467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196394 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:12.196467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196353 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:12.196712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196697 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:12.196910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196873 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.196910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196898 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.196910 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196912 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.197127 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196927 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.197127 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.196940 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fb1ab5-b2bb-421a-add1-07c99039feab-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.198436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.198408 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "kube-api-access-k6stf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:12.198555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.198525 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:12.198859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.198837 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:11:12.198859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.198848 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo" pod "d1fb1ab5-b2bb-421a-add1-07c99039feab" (UID: "d1fb1ab5-b2bb-421a-add1-07c99039feab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:11:12.298889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.298850 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.298889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.298888 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.299162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.298903 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fb1ab5-b2bb-421a-add1-07c99039feab-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.299162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.298923 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6stf\" (UniqueName: \"kubernetes.io/projected/d1fb1ab5-b2bb-421a-add1-07c99039feab-kube-api-access-k6stf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:12.587068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.586978 5932 generic.go:356] "Generic (PLEG): container finished" podID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerID="c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc" exitCode=0 Nov 03 21:11:12.587068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.587028 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" Nov 03 21:11:12.587322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.587071 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" event={"ID":"d1fb1ab5-b2bb-421a-add1-07c99039feab","Type":"ContainerDied","Data":"c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc"} Nov 03 21:11:12.587322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.587128 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb" event={"ID":"d1fb1ab5-b2bb-421a-add1-07c99039feab","Type":"ContainerDied","Data":"e9400cda2b8b5ac89fc4b4d8d2a4a6d917a3f3b60f0d0c2f9e0c04ab283fb966"} Nov 03 21:11:12.587322 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.587157 5932 scope.go:122] "RemoveContainer" containerID="c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc" Nov 03 21:11:12.588982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.588967 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" Nov 03 21:11:12.589166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.588964 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc" event={"ID":"e45f2b2a-be2f-45bb-bcca-1946e91054dd","Type":"ContainerDied","Data":"b97ceea6fda119c587e97406e8a17098bb17fb864cc54792c2e500022e62f4ba"} Nov 03 21:11:12.589317 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.589182 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b97ceea6fda119c587e97406e8a17098bb17fb864cc54792c2e500022e62f4ba" Nov 03 21:11:12.598744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.598726 5932 scope.go:122] "RemoveContainer" containerID="c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc" Nov 03 21:11:12.599005 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:11:12.598977 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc\": container with ID starting with c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc not found: ID does not exist" containerID="c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc" Nov 03 21:11:12.599099 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.599012 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc"} err="failed to get container status \"c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc\": rpc error: code = NotFound desc = could not find container \"c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc\": container with ID starting with c431344deff01ebd3e79b0a8ea03488273f0b397482eddfb90e68d790977eedc not found: ID does not exist" Nov 03 21:11:12.805686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.805651 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb"] Nov 03 21:11:12.810680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:12.810653 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-2a5ec9-openshift-default-64895655d5-xnrlb"] Nov 03 21:11:14.054998 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:14.054956 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" path="/var/lib/kubelet/pods/d1fb1ab5-b2bb-421a-add1-07c99039feab/volumes" Nov 03 21:11:14.177285 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:11:14.177250 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:11:20.658782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:20.658748 5932 generic.go:356] "Generic (PLEG): container finished" podID="660352ac-d467-4113-bbf4-f8b164da8f70" containerID="4f4d4bb9189513dcd920d6df77ad6fe6727741cc8196f1991fa5261bea76cc83" exitCode=0 Nov 03 21:11:20.659229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:20.658825 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" event={"ID":"660352ac-d467-4113-bbf4-f8b164da8f70","Type":"ContainerDied","Data":"4f4d4bb9189513dcd920d6df77ad6fe6727741cc8196f1991fa5261bea76cc83"} Nov 03 21:11:20.659317 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:20.659298 5932 scope.go:122] "RemoveContainer" containerID="4f4d4bb9189513dcd920d6df77ad6fe6727741cc8196f1991fa5261bea76cc83" Nov 03 21:11:20.660933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:20.660902 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:20.661057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:20.660987 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:20.661126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:20.661060 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:21.668538 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:21.668501 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" event={"ID":"660352ac-d467-4113-bbf4-f8b164da8f70","Type":"ContainerStarted","Data":"fa355741cd1f79cc395b04ccccf7baa74ce38fe4c8ea5b0dc3db1a9ee563d7c2"} Nov 03 21:11:21.817567 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:21.817539 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:11:22.502054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:22.502016 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:11:22.503783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:22.503750 5932 patch_prober.go:29] interesting pod/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 21:11:22.504072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:22.504038 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 21:11:23.508058 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:23.508024 5932 patch_prober.go:29] interesting pod/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 21:11:23.508752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:23.508717 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 21:11:24.116147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:24.116108 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg"] Nov 03 21:11:24.116373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:24.116348 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f314596c-tswrg"] Nov 03 21:11:24.501244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:24.501208 5932 patch_prober.go:29] interesting pod/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 21:11:24.501428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:24.501263 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 21:11:25.501721 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:25.501686 5932 patch_prober.go:29] interesting pod/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 21:11:25.502116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:25.501738 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 21:11:26.054938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:26.054906 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43350b94-433a-4a6d-a3bf-989f14f5515e" path="/var/lib/kubelet/pods/43350b94-433a-4a6d-a3bf-989f14f5515e/volumes" Nov 03 21:11:26.502086 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:26.502050 5932 patch_prober.go:29] interesting pod/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 21:11:26.502480 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:26.502108 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 21:11:27.506564 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:27.506532 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:11:27.507612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:27.507590 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:11:40.061683 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.061631 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r"] Nov 03 21:11:40.062393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.062372 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" Nov 03 21:11:40.062393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.062395 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" Nov 03 21:11:40.062554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.062419 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e45f2b2a-be2f-45bb-bcca-1946e91054dd" containerName="revoke-keys" Nov 03 21:11:40.062554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.062424 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45f2b2a-be2f-45bb-bcca-1946e91054dd" containerName="revoke-keys" Nov 03 21:11:40.062554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.062527 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fb1ab5-b2bb-421a-add1-07c99039feab" containerName="istio-proxy" Nov 03 21:11:40.062554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.062541 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45f2b2a-be2f-45bb-bcca-1946e91054dd" containerName="revoke-keys" Nov 03 21:11:40.067553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.067534 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.110993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.110955 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rdp\" (UniqueName: \"kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.111121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.111097 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.111552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.111524 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.141443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.141418 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:11:40.141595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.141478 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:11:40.165987 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.165960 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r"] Nov 03 21:11:40.185253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.185225 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s"] Nov 03 21:11:40.189807 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.189786 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.212330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.212306 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.212459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.212357 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rdp\" (UniqueName: \"kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.212459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.212398 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknpw\" (UniqueName: \"kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.212459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.212421 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.212702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.212550 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.212702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.212692 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.213081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.213054 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.215002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.214977 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.278847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.278816 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s"] Nov 03 21:11:40.301676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.301639 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rdp\" (UniqueName: \"kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp\") pod \"maas-api-revoke-sub-6f89d43d-h858r\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.318040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.317406 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknpw\" (UniqueName: \"kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.318040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.317598 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.318040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.317752 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.320086 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.318658 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.327654 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.326274 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.382740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.382702 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:40.395662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.395607 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknpw\" (UniqueName: \"kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw\") pod \"maas-api-revoke-sub-8d04ac2e-l2n8s\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:40.517846 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:40.512978 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:41.181675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:41.179601 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r"] Nov 03 21:11:41.939918 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:41.939701 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s"] Nov 03 21:11:42.182646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:42.175464 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.107:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:11:42.248778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:42.248739 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 21:11:42.249112 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:42.249086 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:11:42.310953 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:42.310911 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": context deadline exceeded" Nov 03 21:11:42.572358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:42.572321 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" event={"ID":"a4c21127-680d-4632-b165-09d8abffa2f4","Type":"ContainerStarted","Data":"9d8856a7346086cba3231960a1a2363d8925bc20a5e0fe3969bc9d543bab10d4"} Nov 03 21:11:42.637861 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:42.630702 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" event={"ID":"0b69a290-82dd-4133-b4e7-fe9debd670a1","Type":"ContainerStarted","Data":"533044b573b99333e69e365ab5a7628e4780b5dea121a0fa57e785b62c35e356"} Nov 03 21:11:43.652280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:43.652239 5932 generic.go:356] "Generic (PLEG): container finished" podID="a4c21127-680d-4632-b165-09d8abffa2f4" containerID="ca37cb9ae60f0428fa0da016b5435ca10955f970e2c5303780d6424f7ea74a1a" exitCode=0 Nov 03 21:11:43.652769 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:43.652320 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" event={"ID":"a4c21127-680d-4632-b165-09d8abffa2f4","Type":"ContainerDied","Data":"ca37cb9ae60f0428fa0da016b5435ca10955f970e2c5303780d6424f7ea74a1a"} Nov 03 21:11:43.654167 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:43.654135 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" event={"ID":"0b69a290-82dd-4133-b4e7-fe9debd670a1","Type":"ContainerStarted","Data":"78239b3dc147093f50271862b537922a09b5b98364a16be45216bb824ad06f9a"} Nov 03 21:11:43.870930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:43.870866 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" podStartSLOduration=3.8708522419999998 podStartE2EDuration="3.870852242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:11:43.869669259 +0000 UTC m=+2511.934910733" watchObservedRunningTime="2026-11-03 21:11:43.870852242 +0000 UTC m=+2511.936093715" Nov 03 21:11:44.178573 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:11:44.178528 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:11:44.663585 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:44.663550 5932 generic.go:356] "Generic (PLEG): container finished" podID="0b69a290-82dd-4133-b4e7-fe9debd670a1" containerID="78239b3dc147093f50271862b537922a09b5b98364a16be45216bb824ad06f9a" exitCode=0 Nov 03 21:11:44.664233 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:44.664206 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" event={"ID":"0b69a290-82dd-4133-b4e7-fe9debd670a1","Type":"ContainerDied","Data":"78239b3dc147093f50271862b537922a09b5b98364a16be45216bb824ad06f9a"} Nov 03 21:11:44.859998 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:44.859969 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:45.020588 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.020514 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca\") pod \"a4c21127-680d-4632-b165-09d8abffa2f4\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " Nov 03 21:11:45.020588 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.020550 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp\" (UniqueName: \"kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp\") pod \"a4c21127-680d-4632-b165-09d8abffa2f4\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " Nov 03 21:11:45.020859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.020688 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca\") pod \"a4c21127-680d-4632-b165-09d8abffa2f4\" (UID: \"a4c21127-680d-4632-b165-09d8abffa2f4\") " Nov 03 21:11:45.020995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.020948 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca" pod "a4c21127-680d-4632-b165-09d8abffa2f4" (UID: "a4c21127-680d-4632-b165-09d8abffa2f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:45.021370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.021346 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.048788 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.048756 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca" pod "a4c21127-680d-4632-b165-09d8abffa2f4" (UID: "a4c21127-680d-4632-b165-09d8abffa2f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:45.050866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.050834 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp" pod "a4c21127-680d-4632-b165-09d8abffa2f4" (UID: "a4c21127-680d-4632-b165-09d8abffa2f4"). InnerVolumeSpecName "kube-api-access-95rdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:45.137691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.134841 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rdp\" (UniqueName: \"kubernetes.io/projected/a4c21127-680d-4632-b165-09d8abffa2f4-kube-api-access-95rdp\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.137691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.134883 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4c21127-680d-4632-b165-09d8abffa2f4-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.672811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.672779 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" Nov 03 21:11:45.673275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.672819 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r" event={"ID":"a4c21127-680d-4632-b165-09d8abffa2f4","Type":"ContainerDied","Data":"9d8856a7346086cba3231960a1a2363d8925bc20a5e0fe3969bc9d543bab10d4"} Nov 03 21:11:45.673275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.672929 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d8856a7346086cba3231960a1a2363d8925bc20a5e0fe3969bc9d543bab10d4" Nov 03 21:11:45.870212 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.870186 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:45.963057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.962987 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca\") pod \"0b69a290-82dd-4133-b4e7-fe9debd670a1\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " Nov 03 21:11:45.963057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.963033 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw\" (UniqueName: \"kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw\") pod \"0b69a290-82dd-4133-b4e7-fe9debd670a1\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " Nov 03 21:11:45.963323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.963300 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca\") pod \"0b69a290-82dd-4133-b4e7-fe9debd670a1\" (UID: \"0b69a290-82dd-4133-b4e7-fe9debd670a1\") " Nov 03 21:11:45.963645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.963610 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca" pod "0b69a290-82dd-4133-b4e7-fe9debd670a1" (UID: "0b69a290-82dd-4133-b4e7-fe9debd670a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:11:45.963770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.963750 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:45.965121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.965093 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca" pod "0b69a290-82dd-4133-b4e7-fe9debd670a1" (UID: "0b69a290-82dd-4133-b4e7-fe9debd670a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:11:45.965517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:45.965499 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw" pod "0b69a290-82dd-4133-b4e7-fe9debd670a1" (UID: "0b69a290-82dd-4133-b4e7-fe9debd670a1"). InnerVolumeSpecName "kube-api-access-hknpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:11:46.064734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:46.064699 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b69a290-82dd-4133-b4e7-fe9debd670a1-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:46.064734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:46.064731 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknpw\" (UniqueName: \"kubernetes.io/projected/0b69a290-82dd-4133-b4e7-fe9debd670a1-kube-api-access-hknpw\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:11:46.687821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:46.687791 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" Nov 03 21:11:46.688226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:46.687785 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s" event={"ID":"0b69a290-82dd-4133-b4e7-fe9debd670a1","Type":"ContainerDied","Data":"533044b573b99333e69e365ab5a7628e4780b5dea121a0fa57e785b62c35e356"} Nov 03 21:11:46.688226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:46.687944 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="533044b573b99333e69e365ab5a7628e4780b5dea121a0fa57e785b62c35e356" Nov 03 21:11:50.754699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:50.754516 5932 generic.go:356] "Generic (PLEG): container finished" podID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerID="cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441" exitCode=0 Nov 03 21:11:50.755340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:50.755283 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" event={"ID":"c4db7b2e-321b-4631-ad4f-10373b59ae8b","Type":"ContainerDied","Data":"cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441"} Nov 03 21:11:50.756153 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:50.756108 5932 scope.go:122] "RemoveContainer" containerID="cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441" Nov 03 21:11:50.766707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:50.765612 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:50.766707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:50.765746 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:50.766707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:50.765791 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:51.796647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:51.796388 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" event={"ID":"c4db7b2e-321b-4631-ad4f-10373b59ae8b","Type":"ContainerStarted","Data":"98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf"} Nov 03 21:11:51.842646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:51.825113 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:11:51.842646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:51.825261 5932 patch_prober.go:29] interesting pod/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 21:11:51.842646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:51.825318 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 21:11:52.853116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:52.853076 5932 patch_prober.go:29] interesting pod/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 21:11:52.853764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:52.853730 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 21:11:53.836747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:53.836712 5932 patch_prober.go:29] interesting pod/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 21:11:53.837033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:53.837005 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 21:11:54.841034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:54.840995 5932 patch_prober.go:29] interesting pod/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 21:11:54.842370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:54.842336 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 21:11:55.832647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:55.830119 5932 patch_prober.go:29] interesting pod/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 21:11:55.832647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:55.830176 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 21:11:56.824999 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:56.824960 5932 patch_prober.go:29] interesting pod/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Nov 03 21:11:56.825592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:56.825560 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Nov 03 21:11:57.828944 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:57.828906 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:11:57.829350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:57.829324 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:11:57.830205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:57.830179 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:11:59.215330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.215295 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc"] Nov 03 21:11:59.216359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.216334 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0b69a290-82dd-4133-b4e7-fe9debd670a1" containerName="revoke-keys" Nov 03 21:11:59.216432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.216363 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b69a290-82dd-4133-b4e7-fe9debd670a1" containerName="revoke-keys" Nov 03 21:11:59.216489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.216440 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a4c21127-680d-4632-b165-09d8abffa2f4" containerName="revoke-keys" Nov 03 21:11:59.216489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.216452 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c21127-680d-4632-b165-09d8abffa2f4" containerName="revoke-keys" Nov 03 21:11:59.216735 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.216715 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b69a290-82dd-4133-b4e7-fe9debd670a1" containerName="revoke-keys" Nov 03 21:11:59.216828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.216747 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c21127-680d-4632-b165-09d8abffa2f4" containerName="revoke-keys" Nov 03 21:11:59.221346 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.221323 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.316598 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.316569 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.316814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.316706 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.316814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.316746 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cgz4\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.316975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.316874 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.316975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.316917 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.317134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.317109 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.317181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.317170 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.317256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.317242 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.317293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.317276 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.338011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.337985 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc"] Nov 03 21:11:59.418411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418375 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418423 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418476 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418496 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418567 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418654 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418684 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cgz4\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418726 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.418912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.418755 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.419154 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.419051 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.419154 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.419103 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.419276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.419229 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.419361 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.419338 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.419692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.419672 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.421026 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.421005 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.421690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.421661 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.477577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.477519 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.477691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.477617 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cgz4\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4\") pod \"e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.538669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.538606 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:11:59.968208 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.968175 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc"] Nov 03 21:11:59.973533 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:11:59.973504 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23e80b5_7792_4aa0_8ee3_ca98751d70aa.slice/crio-e7e9fece5d3680ebacc3a2cf6a0941a1d1e288e98485e30f144415d7996ef98b WatchSource:0}: Error finding container e7e9fece5d3680ebacc3a2cf6a0941a1d1e288e98485e30f144415d7996ef98b: Status 404 returned error can't find the container with id e7e9fece5d3680ebacc3a2cf6a0941a1d1e288e98485e30f144415d7996ef98b Nov 03 21:11:59.976197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.976161 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:59.976307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.976242 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:11:59.976307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:11:59.976288 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:00.254580 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.254506 5932 scope.go:122] "RemoveContainer" containerID="aa278c2ebf20abf6d328501bb10c744a1d7cf4793ec0caf578263ab65f7f5978" Nov 03 21:12:00.265334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.265312 5932 scope.go:122] "RemoveContainer" containerID="201c92ef65887be06629ca9f76b12ed17448ae3049699f6d14194e5a5af9950a" Nov 03 21:12:00.275703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.275680 5932 scope.go:122] "RemoveContainer" containerID="cac75c4a20ab797f29fe6f3c89a7340d516e8f30b8074d294bd8e17f71bac40a" Nov 03 21:12:00.953031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.953001 5932 generic.go:356] "Generic (PLEG): container finished" podID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerID="16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f" exitCode=0 Nov 03 21:12:00.953297 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.953078 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" event={"ID":"19d01288-f79b-44ca-8aa6-ebad2ce91abe","Type":"ContainerDied","Data":"16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f"} Nov 03 21:12:00.953589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.953555 5932 scope.go:122] "RemoveContainer" containerID="16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f" Nov 03 21:12:00.955054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.955032 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" event={"ID":"f23e80b5-7792-4aa0-8ee3-ca98751d70aa","Type":"ContainerStarted","Data":"f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f"} Nov 03 21:12:00.955162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.955075 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" event={"ID":"f23e80b5-7792-4aa0-8ee3-ca98751d70aa","Type":"ContainerStarted","Data":"e7e9fece5d3680ebacc3a2cf6a0941a1d1e288e98485e30f144415d7996ef98b"} Nov 03 21:12:00.955264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.955239 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:00.955316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.955305 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:00.955355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:00.955336 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:12:01.260706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:01.246571 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podStartSLOduration=2.24655042 podStartE2EDuration="2.24655042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:12:01.246547175 +0000 UTC m=+2529.311788651" watchObservedRunningTime="2026-11-03 21:12:01.24655042 +0000 UTC m=+2529.311791893" Nov 03 21:12:01.467674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:01.464292 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:12:01.545704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:01.545148 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:12:01.547547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:01.547513 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 21:12:01.547699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:01.547568 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 21:12:01.979959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:01.979920 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" event={"ID":"19d01288-f79b-44ca-8aa6-ebad2ce91abe","Type":"ContainerStarted","Data":"080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf"} Nov 03 21:12:02.539647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:02.539561 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 21:12:02.540225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:02.539667 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 21:12:02.705761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:02.705692 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:12:02.707042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:02.707018 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 21:12:02.707220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:02.707201 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 21:12:03.545418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:03.545383 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 21:12:03.546060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:03.546028 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 21:12:03.705770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:03.705737 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 21:12:03.706005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:03.705978 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 21:12:04.545648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:04.545553 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 21:12:04.545648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:04.545611 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 21:12:04.707649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:04.705788 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 21:12:04.707649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:04.705845 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 21:12:05.541548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:05.541508 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Nov 03 21:12:05.541865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:05.541831 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Nov 03 21:12:05.706003 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:05.705969 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 21:12:05.706424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:05.706021 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 21:12:06.553319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:06.553289 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:12:06.553762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:06.553738 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:12:06.554394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:06.554373 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:12:06.705271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:06.705236 5932 patch_prober.go:29] interesting pod/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" start-of-body= Nov 03 21:12:06.705458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:06.705294 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.251:15021/healthz/ready\": dial tcp 10.132.0.251:15021: connect: connection refused" Nov 03 21:12:07.710202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:07.710163 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:12:07.710661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:07.710610 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:12:07.711295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:07.711275 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:12:14.152850 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:12:14.152812 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:12:28.315448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.315412 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs"] Nov 03 21:12:28.319864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.319847 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.370384 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.370352 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:12:28.395617 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.395587 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.395756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.395655 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.395912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.395891 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdr6z\" (UniqueName: \"kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.416149 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.416131 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:12:28.497601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.497570 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdr6z\" (UniqueName: \"kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.497755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.497736 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.497830 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.497773 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.498351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.498331 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.500111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.500093 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.506986 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.506964 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs"] Nov 03 21:12:28.559456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.559434 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdr6z\" (UniqueName: \"kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z\") pod \"maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:28.634761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:28.634738 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:29.012305 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:29.012223 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs"] Nov 03 21:12:29.019000 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:12:29.018973 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f925c37_23ca_495b_81d9_6096aa481255.slice/crio-e3d29c0ae9e9f2fda6ed264c0debb9e0783665b4d56c8356249f235fb9086f29 WatchSource:0}: Error finding container e3d29c0ae9e9f2fda6ed264c0debb9e0783665b4d56c8356249f235fb9086f29: Status 404 returned error can't find the container with id e3d29c0ae9e9f2fda6ed264c0debb9e0783665b4d56c8356249f235fb9086f29 Nov 03 21:12:29.229834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:29.229797 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" event={"ID":"9f925c37-23ca-495b-81d9-6096aa481255","Type":"ContainerStarted","Data":"e3d29c0ae9e9f2fda6ed264c0debb9e0783665b4d56c8356249f235fb9086f29"} Nov 03 21:12:30.239600 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:30.239506 5932 generic.go:356] "Generic (PLEG): container finished" podID="9f925c37-23ca-495b-81d9-6096aa481255" containerID="3dafd483c1171186c47102c8471dd4500c89db49d945bbe964b2c64db1dad245" exitCode=0 Nov 03 21:12:30.239600 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:30.239545 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" event={"ID":"9f925c37-23ca-495b-81d9-6096aa481255","Type":"ContainerDied","Data":"3dafd483c1171186c47102c8471dd4500c89db49d945bbe964b2c64db1dad245"} Nov 03 21:12:31.434675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.434652 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:31.555678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.555571 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z\" (UniqueName: \"kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z\") pod \"9f925c37-23ca-495b-81d9-6096aa481255\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " Nov 03 21:12:31.555865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.555843 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca\") pod \"9f925c37-23ca-495b-81d9-6096aa481255\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " Nov 03 21:12:31.555939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.555916 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca\") pod \"9f925c37-23ca-495b-81d9-6096aa481255\" (UID: \"9f925c37-23ca-495b-81d9-6096aa481255\") " Nov 03 21:12:31.556201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.556177 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca" pod "9f925c37-23ca-495b-81d9-6096aa481255" (UID: "9f925c37-23ca-495b-81d9-6096aa481255"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:31.556376 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.556363 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:31.557574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.557552 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z" pod "9f925c37-23ca-495b-81d9-6096aa481255" (UID: "9f925c37-23ca-495b-81d9-6096aa481255"). InnerVolumeSpecName "kube-api-access-pdr6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:31.557822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.557801 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca" pod "9f925c37-23ca-495b-81d9-6096aa481255" (UID: "9f925c37-23ca-495b-81d9-6096aa481255"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:31.657283 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.657251 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f925c37-23ca-495b-81d9-6096aa481255-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:31.657283 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:31.657278 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdr6z\" (UniqueName: \"kubernetes.io/projected/9f925c37-23ca-495b-81d9-6096aa481255-kube-api-access-pdr6z\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:32.264755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:32.264724 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" Nov 03 21:12:32.264755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:32.264734 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs" event={"ID":"9f925c37-23ca-495b-81d9-6096aa481255","Type":"ContainerDied","Data":"e3d29c0ae9e9f2fda6ed264c0debb9e0783665b4d56c8356249f235fb9086f29"} Nov 03 21:12:32.264957 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:32.264788 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d29c0ae9e9f2fda6ed264c0debb9e0783665b4d56c8356249f235fb9086f29" Nov 03 21:12:35.667478 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.667444 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22"] Nov 03 21:12:35.668122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.668107 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9f925c37-23ca-495b-81d9-6096aa481255" containerName="revoke-keys" Nov 03 21:12:35.668173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.668124 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f925c37-23ca-495b-81d9-6096aa481255" containerName="revoke-keys" Nov 03 21:12:35.668272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.668260 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f925c37-23ca-495b-81d9-6096aa481255" containerName="revoke-keys" Nov 03 21:12:35.671591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.671575 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.705113 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.705089 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.705216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.705173 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b259q\" (UniqueName: \"kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.705354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.705341 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.768683 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.768654 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:12:35.768837 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.768722 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:12:35.774732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.774703 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22"] Nov 03 21:12:35.806521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.806497 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.806683 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.806572 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.806683 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.806610 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b259q\" (UniqueName: \"kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.807265 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.807240 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.808816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.808799 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.863890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.863866 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b259q\" (UniqueName: \"kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q\") pod \"maas-api-revoke-keys-e2e-api-a-8afb98-kvs22\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:35.986019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:35.985966 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:36.367384 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:36.367304 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22"] Nov 03 21:12:36.372411 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:12:36.372377 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b1e8938_4da3_49f4_98df_2f6c424628f6.slice/crio-c9a1e47a54330b04ac9ab8c62a969c3e4195315d1ecc40f378012daf79fdb168 WatchSource:0}: Error finding container c9a1e47a54330b04ac9ab8c62a969c3e4195315d1ecc40f378012daf79fdb168: Status 404 returned error can't find the container with id c9a1e47a54330b04ac9ab8c62a969c3e4195315d1ecc40f378012daf79fdb168 Nov 03 21:12:37.311027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:37.310998 5932 generic.go:356] "Generic (PLEG): container finished" podID="9b1e8938-4da3-49f4-98df-2f6c424628f6" containerID="4a0483c9d568a61451c608231eef17886f65124c9f95db06b350730f6c160dcb" exitCode=0 Nov 03 21:12:37.311448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:37.311034 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" event={"ID":"9b1e8938-4da3-49f4-98df-2f6c424628f6","Type":"ContainerDied","Data":"4a0483c9d568a61451c608231eef17886f65124c9f95db06b350730f6c160dcb"} Nov 03 21:12:37.311448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:37.311102 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" event={"ID":"9b1e8938-4da3-49f4-98df-2f6c424628f6","Type":"ContainerStarted","Data":"c9a1e47a54330b04ac9ab8c62a969c3e4195315d1ecc40f378012daf79fdb168"} Nov 03 21:12:38.476703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.476681 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:38.537791 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.537766 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca\") pod \"9b1e8938-4da3-49f4-98df-2f6c424628f6\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " Nov 03 21:12:38.537930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.537802 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q\" (UniqueName: \"kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q\") pod \"9b1e8938-4da3-49f4-98df-2f6c424628f6\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " Nov 03 21:12:38.537930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.537831 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca\") pod \"9b1e8938-4da3-49f4-98df-2f6c424628f6\" (UID: \"9b1e8938-4da3-49f4-98df-2f6c424628f6\") " Nov 03 21:12:38.538114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.538092 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca" pod "9b1e8938-4da3-49f4-98df-2f6c424628f6" (UID: "9b1e8938-4da3-49f4-98df-2f6c424628f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:38.538461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.538442 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:38.539821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.539793 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q" pod "9b1e8938-4da3-49f4-98df-2f6c424628f6" (UID: "9b1e8938-4da3-49f4-98df-2f6c424628f6"). InnerVolumeSpecName "kube-api-access-b259q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:38.539919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.539897 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca" pod "9b1e8938-4da3-49f4-98df-2f6c424628f6" (UID: "9b1e8938-4da3-49f4-98df-2f6c424628f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:12:38.639494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.639458 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b259q\" (UniqueName: \"kubernetes.io/projected/9b1e8938-4da3-49f4-98df-2f6c424628f6-kube-api-access-b259q\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:38.639494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:38.639491 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b1e8938-4da3-49f4-98df-2f6c424628f6-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:39.327725 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:39.327692 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" Nov 03 21:12:39.327725 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:39.327705 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22" event={"ID":"9b1e8938-4da3-49f4-98df-2f6c424628f6","Type":"ContainerDied","Data":"c9a1e47a54330b04ac9ab8c62a969c3e4195315d1ecc40f378012daf79fdb168"} Nov 03 21:12:39.327955 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:39.327759 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a1e47a54330b04ac9ab8c62a969c3e4195315d1ecc40f378012daf79fdb168" Nov 03 21:12:39.990578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:39.990546 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f"] Nov 03 21:12:40.106033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:40.105996 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g"] Nov 03 21:12:44.152603 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:12:44.152569 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:12:44.991776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:44.991718 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" podUID="5eaa62d1-42a9-414f-8370-276e039fde3b" containerName="payload-processing" containerID="cri-o://dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4" gracePeriod=25 Nov 03 21:12:45.261911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.261858 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:12:45.337488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.337451 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" podUID="33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" containerName="payload-pre-processing" containerID="cri-o://45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd" gracePeriod=25 Nov 03 21:12:45.348638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.348605 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd\" (UniqueName: \"kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd\") pod \"5eaa62d1-42a9-414f-8370-276e039fde3b\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " Nov 03 21:12:45.348743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.348709 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume\") pod \"5eaa62d1-42a9-414f-8370-276e039fde3b\" (UID: \"5eaa62d1-42a9-414f-8370-276e039fde3b\") " Nov 03 21:12:45.349132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.349105 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume" pod "5eaa62d1-42a9-414f-8370-276e039fde3b" (UID: "5eaa62d1-42a9-414f-8370-276e039fde3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:45.349296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.349260 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eaa62d1-42a9-414f-8370-276e039fde3b-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:45.350642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.350603 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd" pod "5eaa62d1-42a9-414f-8370-276e039fde3b" (UID: "5eaa62d1-42a9-414f-8370-276e039fde3b"). InnerVolumeSpecName "kube-api-access-6klbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:45.389073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.389037 5932 generic.go:356] "Generic (PLEG): container finished" podID="5eaa62d1-42a9-414f-8370-276e039fde3b" containerID="dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4" exitCode=0 Nov 03 21:12:45.389201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.389087 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" Nov 03 21:12:45.389201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.389120 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" event={"ID":"5eaa62d1-42a9-414f-8370-276e039fde3b","Type":"ContainerDied","Data":"dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4"} Nov 03 21:12:45.389311 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.389215 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f" event={"ID":"5eaa62d1-42a9-414f-8370-276e039fde3b","Type":"ContainerDied","Data":"389b263660edcf7cae93055c47fb179dea11b2701dd576c76b782a7ca0a59fc1"} Nov 03 21:12:45.389311 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.389249 5932 scope.go:122] "RemoveContainer" containerID="dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4" Nov 03 21:12:45.451279 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.451241 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6klbd\" (UniqueName: \"kubernetes.io/projected/5eaa62d1-42a9-414f-8370-276e039fde3b-kube-api-access-6klbd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:45.451704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.451683 5932 scope.go:122] "RemoveContainer" containerID="dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4" Nov 03 21:12:45.452043 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:12:45.452022 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4\": container with ID starting with dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4 not found: ID does not exist" containerID="dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4" Nov 03 21:12:45.452148 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.452056 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4"} err="failed to get container status \"dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4\": rpc error: code = NotFound desc = could not find container \"dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4\": container with ID starting with dbc29538e0129dedc3e4bb84a27671a9b6a417ad51b14a1fce54d08657fc70a4 not found: ID does not exist" Nov 03 21:12:45.547772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.547709 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f"] Nov 03 21:12:45.550359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.550330 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-7a3593-56b77b4d7f-hxn4f"] Nov 03 21:12:45.611001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.610977 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:12:45.755193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.755159 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr\" (UniqueName: \"kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr\") pod \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " Nov 03 21:12:45.755363 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.755323 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume\") pod \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\" (UID: \"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3\") " Nov 03 21:12:45.755811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.755780 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume" pod "33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" (UID: "33ac0eef-b9ac-4df3-a6e7-abb72fe741e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:45.755965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.755946 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:45.757316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.757292 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr" pod "33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" (UID: "33ac0eef-b9ac-4df3-a6e7-abb72fe741e3"). InnerVolumeSpecName "kube-api-access-7hthr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:45.876699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:45.871059 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hthr\" (UniqueName: \"kubernetes.io/projected/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3-kube-api-access-7hthr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:46.100392 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.097304 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eaa62d1-42a9-414f-8370-276e039fde3b" path="/var/lib/kubelet/pods/5eaa62d1-42a9-414f-8370-276e039fde3b/volumes" Nov 03 21:12:46.398703 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.398672 5932 generic.go:356] "Generic (PLEG): container finished" podID="33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" containerID="45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd" exitCode=0 Nov 03 21:12:46.399171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.398714 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" Nov 03 21:12:46.399171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.398754 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" event={"ID":"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3","Type":"ContainerDied","Data":"45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd"} Nov 03 21:12:46.399171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.398816 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g" event={"ID":"33ac0eef-b9ac-4df3-a6e7-abb72fe741e3","Type":"ContainerDied","Data":"2ce77033ebf529f53ed01969c7d05985503b0c07bc51a839dc61268dee5c337b"} Nov 03 21:12:46.399171 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.398850 5932 scope.go:122] "RemoveContainer" containerID="45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd" Nov 03 21:12:46.410467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.410451 5932 scope.go:122] "RemoveContainer" containerID="45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd" Nov 03 21:12:46.410784 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:12:46.410763 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd\": container with ID starting with 45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd not found: ID does not exist" containerID="45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd" Nov 03 21:12:46.410873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.410791 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd"} err="failed to get container status \"45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd\": rpc error: code = NotFound desc = could not find container \"45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd\": container with ID starting with 45d221ac8c249170a295b2841b194386ee4d6943bc346775dd7912d184371ddd not found: ID does not exist" Nov 03 21:12:46.553435 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.553404 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g"] Nov 03 21:12:46.574146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:46.574122 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-7a3593-67cb4bddb7-x8h9g"] Nov 03 21:12:47.296659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:47.296607 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw"] Nov 03 21:12:47.385403 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:47.385358 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67"] Nov 03 21:12:47.928039 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:47.928004 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc"] Nov 03 21:12:47.932876 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:47.932847 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1e6fe1-vqgmc"] Nov 03 21:12:48.052945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:48.052913 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" path="/var/lib/kubelet/pods/33ac0eef-b9ac-4df3-a6e7-abb72fe741e3/volumes" Nov 03 21:12:48.053396 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:48.053382 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45f2b2a-be2f-45bb-bcca-1946e91054dd" path="/var/lib/kubelet/pods/e45f2b2a-be2f-45bb-bcca-1946e91054dd/volumes" Nov 03 21:12:52.297666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.297611 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" podUID="f318cf3f-313b-41f7-a5e4-6465fdd036ed" containerName="payload-processing" containerID="cri-o://32a9effd3569c3b76f637ea5da16cbb26a26d632835e2fa7ebd5524c5f217ff6" gracePeriod=25 Nov 03 21:12:52.387016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.386438 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" podUID="f19d6694-9e7a-421e-9280-1836fee697c1" containerName="payload-pre-processing" containerID="cri-o://8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c" gracePeriod=25 Nov 03 21:12:52.458719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.458679 5932 generic.go:356] "Generic (PLEG): container finished" podID="f318cf3f-313b-41f7-a5e4-6465fdd036ed" containerID="32a9effd3569c3b76f637ea5da16cbb26a26d632835e2fa7ebd5524c5f217ff6" exitCode=0 Nov 03 21:12:52.458849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.458717 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" event={"ID":"f318cf3f-313b-41f7-a5e4-6465fdd036ed","Type":"ContainerDied","Data":"32a9effd3569c3b76f637ea5da16cbb26a26d632835e2fa7ebd5524c5f217ff6"} Nov 03 21:12:52.673138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.673113 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:12:52.706594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.706562 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:12:52.778552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.778525 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume\") pod \"f19d6694-9e7a-421e-9280-1836fee697c1\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " Nov 03 21:12:52.778735 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.778722 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5\" (UniqueName: \"kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5\") pod \"f19d6694-9e7a-421e-9280-1836fee697c1\" (UID: \"f19d6694-9e7a-421e-9280-1836fee697c1\") " Nov 03 21:12:52.778796 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.778748 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume\") pod \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " Nov 03 21:12:52.778796 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.778779 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2\" (UniqueName: \"kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2\") pod \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\" (UID: \"f318cf3f-313b-41f7-a5e4-6465fdd036ed\") " Nov 03 21:12:52.778952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.778927 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume" pod "f19d6694-9e7a-421e-9280-1836fee697c1" (UID: "f19d6694-9e7a-421e-9280-1836fee697c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:52.779107 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.779081 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume" pod "f318cf3f-313b-41f7-a5e4-6465fdd036ed" (UID: "f318cf3f-313b-41f7-a5e4-6465fdd036ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:12:52.779344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.779325 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f318cf3f-313b-41f7-a5e4-6465fdd036ed-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:52.779397 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.779347 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f19d6694-9e7a-421e-9280-1836fee697c1-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:52.780960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.780937 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2" pod "f318cf3f-313b-41f7-a5e4-6465fdd036ed" (UID: "f318cf3f-313b-41f7-a5e4-6465fdd036ed"). InnerVolumeSpecName "kube-api-access-g67p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:52.781043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.781002 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5" pod "f19d6694-9e7a-421e-9280-1836fee697c1" (UID: "f19d6694-9e7a-421e-9280-1836fee697c1"). InnerVolumeSpecName "kube-api-access-8ljd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:12:52.883099 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.883063 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljd5\" (UniqueName: \"kubernetes.io/projected/f19d6694-9e7a-421e-9280-1836fee697c1-kube-api-access-8ljd5\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:52.883272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:52.883103 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g67p2\" (UniqueName: \"kubernetes.io/projected/f318cf3f-313b-41f7-a5e4-6465fdd036ed-kube-api-access-g67p2\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:12:53.468844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.468810 5932 generic.go:356] "Generic (PLEG): container finished" podID="f19d6694-9e7a-421e-9280-1836fee697c1" containerID="8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c" exitCode=0 Nov 03 21:12:53.469352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.468874 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" Nov 03 21:12:53.469352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.468892 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" event={"ID":"f19d6694-9e7a-421e-9280-1836fee697c1","Type":"ContainerDied","Data":"8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c"} Nov 03 21:12:53.469352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.468955 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67" event={"ID":"f19d6694-9e7a-421e-9280-1836fee697c1","Type":"ContainerDied","Data":"9d88eabad08edb7577349b40e6fb3808a780821fdd7483dec40dadcde654137f"} Nov 03 21:12:53.469352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.468987 5932 scope.go:122] "RemoveContainer" containerID="8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c" Nov 03 21:12:53.470500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.470483 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" Nov 03 21:12:53.470604 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.470498 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw" event={"ID":"f318cf3f-313b-41f7-a5e4-6465fdd036ed","Type":"ContainerDied","Data":"e7f945450939a33578647c9817463b57a069946e50d1622e2283b17464805ddd"} Nov 03 21:12:53.483198 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.483178 5932 scope.go:122] "RemoveContainer" containerID="8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c" Nov 03 21:12:53.483506 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:12:53.483488 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c\": container with ID starting with 8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c not found: ID does not exist" containerID="8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c" Nov 03 21:12:53.483558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.483513 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c"} err="failed to get container status \"8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c\": rpc error: code = NotFound desc = could not find container \"8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c\": container with ID starting with 8278c04b8a9105a0cf64484598fd7dc01f3e37e29eec5a85139862ff20edf37c not found: ID does not exist" Nov 03 21:12:53.483558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.483529 5932 scope.go:122] "RemoveContainer" containerID="32a9effd3569c3b76f637ea5da16cbb26a26d632835e2fa7ebd5524c5f217ff6" Nov 03 21:12:53.625948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.625923 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw"] Nov 03 21:12:53.633298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.633275 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8afb98-75854f9bbc-rmmfw"] Nov 03 21:12:53.776058 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.776021 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67"] Nov 03 21:12:53.780866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:53.780837 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8afb98-5ccfc65695-psr67"] Nov 03 21:12:54.054136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:54.054062 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19d6694-9e7a-421e-9280-1836fee697c1" path="/var/lib/kubelet/pods/f19d6694-9e7a-421e-9280-1836fee697c1/volumes" Nov 03 21:12:54.054521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:12:54.054508 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f318cf3f-313b-41f7-a5e4-6465fdd036ed" path="/var/lib/kubelet/pods/f318cf3f-313b-41f7-a5e4-6465fdd036ed/volumes" Nov 03 21:13:00.321249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:00.321222 5932 scope.go:122] "RemoveContainer" containerID="2c9f18bc0c6aa9d51aafac74985a2c70f46c4eb71271405ffe7b6a8aaf2688d4" Nov 03 21:13:02.797473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.797440 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp"] Nov 03 21:13:02.798088 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798071 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f19d6694-9e7a-421e-9280-1836fee697c1" containerName="payload-pre-processing" Nov 03 21:13:02.798131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798088 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19d6694-9e7a-421e-9280-1836fee697c1" containerName="payload-pre-processing" Nov 03 21:13:02.798131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798124 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f318cf3f-313b-41f7-a5e4-6465fdd036ed" containerName="payload-processing" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798131 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f318cf3f-313b-41f7-a5e4-6465fdd036ed" containerName="payload-processing" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798138 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5eaa62d1-42a9-414f-8370-276e039fde3b" containerName="payload-processing" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798143 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eaa62d1-42a9-414f-8370-276e039fde3b" containerName="payload-processing" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798152 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9b1e8938-4da3-49f4-98df-2f6c424628f6" containerName="revoke-keys" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798158 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b1e8938-4da3-49f4-98df-2f6c424628f6" containerName="revoke-keys" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798170 5932 cpu_manager.go:414] "containerMap: removing container" podUID="33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" containerName="payload-pre-processing" Nov 03 21:13:02.798215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798175 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" containerName="payload-pre-processing" Nov 03 21:13:02.798437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798255 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eaa62d1-42a9-414f-8370-276e039fde3b" containerName="payload-processing" Nov 03 21:13:02.798437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798265 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b1e8938-4da3-49f4-98df-2f6c424628f6" containerName="revoke-keys" Nov 03 21:13:02.798437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798274 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ac0eef-b9ac-4df3-a6e7-abb72fe741e3" containerName="payload-pre-processing" Nov 03 21:13:02.798437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798283 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19d6694-9e7a-421e-9280-1836fee697c1" containerName="payload-pre-processing" Nov 03 21:13:02.798437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.798295 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f318cf3f-313b-41f7-a5e4-6465fdd036ed" containerName="payload-processing" Nov 03 21:13:02.803466 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.803448 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:02.900098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.900068 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c971dc-serving-cert\"" Nov 03 21:13:02.904498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.904473 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp"] Nov 03 21:13:02.939959 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.939931 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:02.940052 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.939981 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77vdr\" (UniqueName: \"kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:02.940108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.940089 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:02.940146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:02.940118 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.041267 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.041236 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.041428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.041300 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77vdr\" (UniqueName: \"kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.041428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.041359 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.041428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.041380 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.041579 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:03.041379 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-serving-cert: secret "maas-api-e2e-inf-b-c971dc-serving-cert" not found Nov 03 21:13:03.041579 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:03.041572 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls podName:378e125d-9e59-412d-9337-9c29b73b5d9d nodeName:}" failed. No retries permitted until 2026-11-03 21:13:03.54155133 +0000 UTC m=+2591.606792788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls") pod "maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" (UID: "378e125d-9e59-412d-9337-9c29b73b5d9d") : secret "maas-api-e2e-inf-b-c971dc-serving-cert" not found Nov 03 21:13:03.042154 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.042135 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.043824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.043804 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.102927 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.102854 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77vdr\" (UniqueName: \"kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.566211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.566177 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.568660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.568636 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\") pod \"maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:03.718047 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:03.718022 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:04.106809 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:04.106779 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp"] Nov 03 21:13:04.113061 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:13:04.113034 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378e125d_9e59_412d_9337_9c29b73b5d9d.slice/crio-993a95c305a84937a1d05033d37e3b893e13ba6045fe0b58e21446cc5b170221 WatchSource:0}: Error finding container 993a95c305a84937a1d05033d37e3b893e13ba6045fe0b58e21446cc5b170221: Status 404 returned error can't find the container with id 993a95c305a84937a1d05033d37e3b893e13ba6045fe0b58e21446cc5b170221 Nov 03 21:13:04.114777 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:04.114756 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:13:04.556300 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:04.556269 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" event={"ID":"378e125d-9e59-412d-9337-9c29b73b5d9d","Type":"ContainerStarted","Data":"993a95c305a84937a1d05033d37e3b893e13ba6045fe0b58e21446cc5b170221"} Nov 03 21:13:05.049121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:05.049093 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw"] Nov 03 21:13:05.095461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:05.095429 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d150fa62-gs6nw"] Nov 03 21:13:05.565679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:05.565645 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" event={"ID":"378e125d-9e59-412d-9337-9c29b73b5d9d","Type":"ContainerStarted","Data":"b96209bba11551fcbb44179b64e60beb092dd625505c8b7e8582d8b4bbd5a1fe"} Nov 03 21:13:05.566052 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:05.565730 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:05.679779 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:05.679732 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" podStartSLOduration=2.857360079 podStartE2EDuration="3.679719575s" totalImagesPullingTime="822.359496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:05.675530314 +0000 UTC m=+2593.740771788" watchObservedRunningTime="2026-11-03 21:13:05.679719575 +0000 UTC m=+2593.744961045" Nov 03 21:13:06.073027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:06.072988 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1dd41dc-505a-4b99-83e5-f2f9be4db6aa" path="/var/lib/kubelet/pods/f1dd41dc-505a-4b99-83e5-f2f9be4db6aa/volumes" Nov 03 21:13:11.579076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:11.579042 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:13:14.151117 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:14.151077 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:13:37.386877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.386670 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf"] Nov 03 21:13:37.387345 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.386995 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" podUID="735ea8bc-59ec-4b56-b1cb-fad48cc98675" containerName="maas-api" containerID="cri-o://73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce" gracePeriod=30 Nov 03 21:13:37.713132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.713105 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:13:37.829792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.828705 5932 generic.go:356] "Generic (PLEG): container finished" podID="735ea8bc-59ec-4b56-b1cb-fad48cc98675" containerID="73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce" exitCode=0 Nov 03 21:13:37.829792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.829224 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" Nov 03 21:13:37.829792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.829382 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" event={"ID":"735ea8bc-59ec-4b56-b1cb-fad48cc98675","Type":"ContainerDied","Data":"73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce"} Nov 03 21:13:37.829792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.829433 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf" event={"ID":"735ea8bc-59ec-4b56-b1cb-fad48cc98675","Type":"ContainerDied","Data":"491d33261c3a4b199dc278bf2662ab28cf1263ff1cdc00fbaa3895a2ebe6d86e"} Nov 03 21:13:37.829792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.829467 5932 scope.go:122] "RemoveContainer" containerID="73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce" Nov 03 21:13:37.845120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.844952 5932 scope.go:122] "RemoveContainer" containerID="73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce" Nov 03 21:13:37.845332 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:37.845308 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce\": container with ID starting with 73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce not found: ID does not exist" containerID="73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce" Nov 03 21:13:37.845398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.845345 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce"} err="failed to get container status \"73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce\": rpc error: code = NotFound desc = could not find container \"73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce\": container with ID starting with 73299442059f7b038f0ca515d35ca7bf33b633fd826ce9ff3e507d281f60a6ce not found: ID does not exist" Nov 03 21:13:37.856886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.856855 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca\") pod \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " Nov 03 21:13:37.857048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.856924 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\") pod \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " Nov 03 21:13:37.857048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.856992 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw\" (UniqueName: \"kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw\") pod \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " Nov 03 21:13:37.857212 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.857193 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs\") pod \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\" (UID: \"735ea8bc-59ec-4b56-b1cb-fad48cc98675\") " Nov 03 21:13:37.857276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.857225 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca" pod "735ea8bc-59ec-4b56-b1cb-fad48cc98675" (UID: "735ea8bc-59ec-4b56-b1cb-fad48cc98675"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:37.858235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.858209 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/735ea8bc-59ec-4b56-b1cb-fad48cc98675-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:37.859805 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.859766 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs" pod "735ea8bc-59ec-4b56-b1cb-fad48cc98675" (UID: "735ea8bc-59ec-4b56-b1cb-fad48cc98675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:37.859805 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.859785 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls" pod "735ea8bc-59ec-4b56-b1cb-fad48cc98675" (UID: "735ea8bc-59ec-4b56-b1cb-fad48cc98675"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:37.859990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.859860 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw" pod "735ea8bc-59ec-4b56-b1cb-fad48cc98675" (UID: "735ea8bc-59ec-4b56-b1cb-fad48cc98675"). InnerVolumeSpecName "kube-api-access-cv2hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:37.971035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.966229 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2hw\" (UniqueName: \"kubernetes.io/projected/735ea8bc-59ec-4b56-b1cb-fad48cc98675-kube-api-access-cv2hw\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:37.971035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.966276 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:37.971035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:37.966292 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/735ea8bc-59ec-4b56-b1cb-fad48cc98675-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:38.238697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:38.238512 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf"] Nov 03 21:13:38.284607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:38.284573 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a3593-694f4f5589-vhhlf"] Nov 03 21:13:40.053647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:40.053586 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="735ea8bc-59ec-4b56-b1cb-fad48cc98675" path="/var/lib/kubelet/pods/735ea8bc-59ec-4b56-b1cb-fad48cc98675/volumes" Nov 03 21:13:40.164444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:40.164408 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs"] Nov 03 21:13:40.170153 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:40.170125 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7a3593-vq7cs"] Nov 03 21:13:42.054388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:42.054357 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f925c37-23ca-495b-81d9-6096aa481255" path="/var/lib/kubelet/pods/9f925c37-23ca-495b-81d9-6096aa481255/volumes" Nov 03 21:13:44.054420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:44.054389 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6"] Nov 03 21:13:44.095420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:44.095392 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e76ad77a-vtzt6"] Nov 03 21:13:44.151356 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:44.151323 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:13:46.051937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:46.051902 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764" path="/var/lib/kubelet/pods/389f9a4c-7b7a-44c6-9ac6-c1c93fbdd764/volumes" Nov 03 21:13:46.911134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:46.911101 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf"] Nov 03 21:13:46.911337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:46.911314 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" containerID="cri-o://3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76" gracePeriod=30 Nov 03 21:13:47.856204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.856166 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk"] Nov 03 21:13:47.856838 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.856822 5932 cpu_manager.go:414] "containerMap: removing container" podUID="735ea8bc-59ec-4b56-b1cb-fad48cc98675" containerName="maas-api" Nov 03 21:13:47.856895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.856839 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="735ea8bc-59ec-4b56-b1cb-fad48cc98675" containerName="maas-api" Nov 03 21:13:47.856981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.856971 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="735ea8bc-59ec-4b56-b1cb-fad48cc98675" containerName="maas-api" Nov 03 21:13:47.862080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.862063 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:47.915371 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.915340 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk"] Nov 03 21:13:47.920237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.920210 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:47.920358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.920245 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:47.920405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.920398 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465vm\" (UniqueName: \"kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:47.956450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.956423 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:13:47.956590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:47.956454 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:13:48.021930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.021895 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465vm\" (UniqueName: \"kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.022101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.022002 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.022101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.022036 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.022663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.022609 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.024438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.024411 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.079085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.079062 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-465vm\" (UniqueName: \"kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm\") pod \"maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.176246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.176218 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:48.555807 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.555732 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk"] Nov 03 21:13:48.561311 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:13:48.561283 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cccdaf9_5035_41b5_82e2_237ce524f18c.slice/crio-a1de9f9f5cdc3abea27373af8d09321fa1aa2f50a76d8b4b63786bee6f8b81ef WatchSource:0}: Error finding container a1de9f9f5cdc3abea27373af8d09321fa1aa2f50a76d8b4b63786bee6f8b81ef: Status 404 returned error can't find the container with id a1de9f9f5cdc3abea27373af8d09321fa1aa2f50a76d8b4b63786bee6f8b81ef Nov 03 21:13:48.917238 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.917200 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" event={"ID":"8cccdaf9-5035-41b5-82e2-237ce524f18c","Type":"ContainerStarted","Data":"e0da914476a95fa295f360017fae838983ba9fc142d72b23a3390cd11d8092ed"} Nov 03 21:13:48.917599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:48.917261 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" event={"ID":"8cccdaf9-5035-41b5-82e2-237ce524f18c","Type":"ContainerStarted","Data":"a1de9f9f5cdc3abea27373af8d09321fa1aa2f50a76d8b4b63786bee6f8b81ef"} Nov 03 21:13:49.028586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:49.028537 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" podStartSLOduration=2.028523838 podStartE2EDuration="2.028523838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:13:49.02444774 +0000 UTC m=+2637.089689212" watchObservedRunningTime="2026-11-03 21:13:49.028523838 +0000 UTC m=+2637.093765311" Nov 03 21:13:49.962177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:49.962136 5932 generic.go:356] "Generic (PLEG): container finished" podID="8cccdaf9-5035-41b5-82e2-237ce524f18c" containerID="e0da914476a95fa295f360017fae838983ba9fc142d72b23a3390cd11d8092ed" exitCode=0 Nov 03 21:13:49.962605 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:49.962222 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" event={"ID":"8cccdaf9-5035-41b5-82e2-237ce524f18c","Type":"ContainerDied","Data":"e0da914476a95fa295f360017fae838983ba9fc142d72b23a3390cd11d8092ed"} Nov 03 21:13:51.133551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.133522 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:51.272921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.272841 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm\" (UniqueName: \"kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm\") pod \"8cccdaf9-5035-41b5-82e2-237ce524f18c\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " Nov 03 21:13:51.272921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.272887 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca\") pod \"8cccdaf9-5035-41b5-82e2-237ce524f18c\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " Nov 03 21:13:51.273201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.273040 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca\") pod \"8cccdaf9-5035-41b5-82e2-237ce524f18c\" (UID: \"8cccdaf9-5035-41b5-82e2-237ce524f18c\") " Nov 03 21:13:51.273488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.273459 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca" pod "8cccdaf9-5035-41b5-82e2-237ce524f18c" (UID: "8cccdaf9-5035-41b5-82e2-237ce524f18c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:51.273726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.273701 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.275247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.275220 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca" pod "8cccdaf9-5035-41b5-82e2-237ce524f18c" (UID: "8cccdaf9-5035-41b5-82e2-237ce524f18c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:13:51.275357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.275248 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm" pod "8cccdaf9-5035-41b5-82e2-237ce524f18c" (UID: "8cccdaf9-5035-41b5-82e2-237ce524f18c"). InnerVolumeSpecName "kube-api-access-465vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:51.375264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.375229 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-465vm\" (UniqueName: \"kubernetes.io/projected/8cccdaf9-5035-41b5-82e2-237ce524f18c-kube-api-access-465vm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.375264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.375260 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cccdaf9-5035-41b5-82e2-237ce524f18c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:51.978921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.978897 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" Nov 03 21:13:51.979061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.978935 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk" event={"ID":"8cccdaf9-5035-41b5-82e2-237ce524f18c","Type":"ContainerDied","Data":"a1de9f9f5cdc3abea27373af8d09321fa1aa2f50a76d8b4b63786bee6f8b81ef"} Nov 03 21:13:51.979061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:51.978980 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1de9f9f5cdc3abea27373af8d09321fa1aa2f50a76d8b4b63786bee6f8b81ef" Nov 03 21:13:52.189130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.189059 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:13:52.284323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284297 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.284490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284371 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.284490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284402 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.284490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284429 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.284728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284573 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.284728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284608 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.284728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284687 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.284728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284703 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.284728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284700 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.285023 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284767 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.285023 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284845 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.285023 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.284861 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket\") pod \"486253bb-d0a1-4d28-a087-c132a05238ea\" (UID: \"486253bb-d0a1-4d28-a087-c132a05238ea\") " Nov 03 21:13:52.285217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285136 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.285217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285161 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:13:52.285387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285368 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.285458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285388 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.285458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285398 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.285458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285406 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486253bb-d0a1-4d28-a087-c132a05238ea-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.285458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.285415 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.286660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.286630 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:13:52.286811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.286786 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:52.286888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.286809 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "kube-api-access-x6gcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:13:52.287121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.287105 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy" pod "486253bb-d0a1-4d28-a087-c132a05238ea" (UID: "486253bb-d0a1-4d28-a087-c132a05238ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:13:52.387298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.387262 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486253bb-d0a1-4d28-a087-c132a05238ea-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.387298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.387300 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.387486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.387313 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6gcm\" (UniqueName: \"kubernetes.io/projected/486253bb-d0a1-4d28-a087-c132a05238ea-kube-api-access-x6gcm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.387486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.387323 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486253bb-d0a1-4d28-a087-c132a05238ea-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:13:52.988146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.988110 5932 generic.go:356] "Generic (PLEG): container finished" podID="486253bb-d0a1-4d28-a087-c132a05238ea" containerID="3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76" exitCode=0 Nov 03 21:13:52.988329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.988190 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" Nov 03 21:13:52.988329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.988198 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" event={"ID":"486253bb-d0a1-4d28-a087-c132a05238ea","Type":"ContainerDied","Data":"3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76"} Nov 03 21:13:52.988329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.988276 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf" event={"ID":"486253bb-d0a1-4d28-a087-c132a05238ea","Type":"ContainerDied","Data":"fcf2cfea89e547cbc71ac45d90bbd92aad25c09e7dd46c317cf41b3abad4b461"} Nov 03 21:13:52.988329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:52.988297 5932 scope.go:122] "RemoveContainer" containerID="3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76" Nov 03 21:13:53.000868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.000847 5932 scope.go:122] "RemoveContainer" containerID="b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe" Nov 03 21:13:53.011822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.011803 5932 scope.go:122] "RemoveContainer" containerID="3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76" Nov 03 21:13:53.012110 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:53.012088 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76\": container with ID starting with 3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76 not found: ID does not exist" containerID="3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76" Nov 03 21:13:53.012195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.012117 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76"} err="failed to get container status \"3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76\": rpc error: code = NotFound desc = could not find container \"3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76\": container with ID starting with 3d0f346364391e8b7138e0200df4ef8c6823a09f964557229df97cab1e851f76 not found: ID does not exist" Nov 03 21:13:53.012195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.012134 5932 scope.go:122] "RemoveContainer" containerID="b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe" Nov 03 21:13:53.012385 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:13:53.012362 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe\": container with ID starting with b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe not found: ID does not exist" containerID="b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe" Nov 03 21:13:53.012446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.012393 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe"} err="failed to get container status \"b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe\": rpc error: code = NotFound desc = could not find container \"b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe\": container with ID starting with b4ae0a37d87dffd5d9e8bad8b5169bee36229ae8c9f50c245b5c4cfc7da4cffe not found: ID does not exist" Nov 03 21:13:53.190338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.190303 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf"] Nov 03 21:13:53.236343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:53.236307 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7a3593-openshift-default-7bb997ffd5-4v2vf"] Nov 03 21:13:54.052120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:54.052083 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" path="/var/lib/kubelet/pods/486253bb-d0a1-4d28-a087-c132a05238ea/volumes" Nov 03 21:13:59.360354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:59.360316 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk"] Nov 03 21:13:59.463750 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:13:59.463719 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm"] Nov 03 21:14:04.361130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.361082 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" podUID="844b4c14-d200-405c-816b-9297609dbe3d" containerName="payload-processing" containerID="cri-o://1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01" gracePeriod=25 Nov 03 21:14:04.464139 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.464084 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" podUID="c207bcd6-8186-4416-b183-d806d5ff6f28" containerName="payload-pre-processing" containerID="cri-o://d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3" gracePeriod=25 Nov 03 21:14:04.731314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.731289 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:14:04.770343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.770310 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8\" (UniqueName: \"kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8\") pod \"844b4c14-d200-405c-816b-9297609dbe3d\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " Nov 03 21:14:04.770495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.770395 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume\") pod \"844b4c14-d200-405c-816b-9297609dbe3d\" (UID: \"844b4c14-d200-405c-816b-9297609dbe3d\") " Nov 03 21:14:04.771398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.771361 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume" pod "844b4c14-d200-405c-816b-9297609dbe3d" (UID: "844b4c14-d200-405c-816b-9297609dbe3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:04.778421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.778396 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8" pod "844b4c14-d200-405c-816b-9297609dbe3d" (UID: "844b4c14-d200-405c-816b-9297609dbe3d"). InnerVolumeSpecName "kube-api-access-pc6m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:04.840932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.840907 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:14:04.871459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.871417 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume\") pod \"c207bcd6-8186-4416-b183-d806d5ff6f28\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " Nov 03 21:14:04.871601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.871550 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28\" (UniqueName: \"kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28\") pod \"c207bcd6-8186-4416-b183-d806d5ff6f28\" (UID: \"c207bcd6-8186-4416-b183-d806d5ff6f28\") " Nov 03 21:14:04.871824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.871798 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume" pod "c207bcd6-8186-4416-b183-d806d5ff6f28" (UID: "c207bcd6-8186-4416-b183-d806d5ff6f28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:04.872248 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.872226 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c207bcd6-8186-4416-b183-d806d5ff6f28-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:04.872347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.872253 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc6m8\" (UniqueName: \"kubernetes.io/projected/844b4c14-d200-405c-816b-9297609dbe3d-kube-api-access-pc6m8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:04.872347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.872269 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844b4c14-d200-405c-816b-9297609dbe3d-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:04.873911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.873880 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28" pod "c207bcd6-8186-4416-b183-d806d5ff6f28" (UID: "c207bcd6-8186-4416-b183-d806d5ff6f28"). InnerVolumeSpecName "kube-api-access-5dp28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:04.973461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:04.973384 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dp28\" (UniqueName: \"kubernetes.io/projected/c207bcd6-8186-4416-b183-d806d5ff6f28-kube-api-access-5dp28\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:05.082821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.082792 5932 generic.go:356] "Generic (PLEG): container finished" podID="844b4c14-d200-405c-816b-9297609dbe3d" containerID="1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01" exitCode=0 Nov 03 21:14:05.083011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.082840 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" Nov 03 21:14:05.083011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.082870 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" event={"ID":"844b4c14-d200-405c-816b-9297609dbe3d","Type":"ContainerDied","Data":"1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01"} Nov 03 21:14:05.083011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.082926 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk" event={"ID":"844b4c14-d200-405c-816b-9297609dbe3d","Type":"ContainerDied","Data":"1df517f4d44f39c1821f3244a84df5ef28aedd2555985293336eb2de9547479b"} Nov 03 21:14:05.083011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.082952 5932 scope.go:122] "RemoveContainer" containerID="1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01" Nov 03 21:14:05.084192 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.084171 5932 generic.go:356] "Generic (PLEG): container finished" podID="c207bcd6-8186-4416-b183-d806d5ff6f28" containerID="d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3" exitCode=0 Nov 03 21:14:05.084321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.084218 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" Nov 03 21:14:05.084321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.084270 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" event={"ID":"c207bcd6-8186-4416-b183-d806d5ff6f28","Type":"ContainerDied","Data":"d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3"} Nov 03 21:14:05.084321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.084310 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm" event={"ID":"c207bcd6-8186-4416-b183-d806d5ff6f28","Type":"ContainerDied","Data":"4a6e8a384bd64a4969db9bb7ea7e8bdb86c78f1345526a9f56e2ddecc8d44c25"} Nov 03 21:14:05.096314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.096298 5932 scope.go:122] "RemoveContainer" containerID="1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01" Nov 03 21:14:05.096565 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:14:05.096541 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01\": container with ID starting with 1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01 not found: ID does not exist" containerID="1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01" Nov 03 21:14:05.096647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.096576 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01"} err="failed to get container status \"1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01\": rpc error: code = NotFound desc = could not find container \"1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01\": container with ID starting with 1c6c98df367b61e122a88bfde5781c6ec2aa7d2454ef178c01d4b2db8d882a01 not found: ID does not exist" Nov 03 21:14:05.096647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.096593 5932 scope.go:122] "RemoveContainer" containerID="d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3" Nov 03 21:14:05.106279 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.106257 5932 scope.go:122] "RemoveContainer" containerID="d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3" Nov 03 21:14:05.106552 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:14:05.106533 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3\": container with ID starting with d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3 not found: ID does not exist" containerID="d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3" Nov 03 21:14:05.106634 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.106559 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3"} err="failed to get container status \"d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3\": rpc error: code = NotFound desc = could not find container \"d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3\": container with ID starting with d4a1d5a39939133d3367acdfb96bd9fd78ee9506500c59ea860ff288ecab52a3 not found: ID does not exist" Nov 03 21:14:05.240754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.240599 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk"] Nov 03 21:14:05.243764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.243736 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-10fa59-7db8d9ccf7-l66gk"] Nov 03 21:14:05.344979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.344901 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm"] Nov 03 21:14:05.348767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:05.348738 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-10fa59-7f6c64d95-24bpm"] Nov 03 21:14:06.052056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:06.052023 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844b4c14-d200-405c-816b-9297609dbe3d" path="/var/lib/kubelet/pods/844b4c14-d200-405c-816b-9297609dbe3d/volumes" Nov 03 21:14:06.052497 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:06.052481 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c207bcd6-8186-4416-b183-d806d5ff6f28" path="/var/lib/kubelet/pods/c207bcd6-8186-4416-b183-d806d5ff6f28/volumes" Nov 03 21:14:12.133667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:12.133613 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22"] Nov 03 21:14:12.179751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:12.179717 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-8afb98-kvs22"] Nov 03 21:14:14.053342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:14.053309 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b1e8938-4da3-49f4-98df-2f6c424628f6" path="/var/lib/kubelet/pods/9b1e8938-4da3-49f4-98df-2f6c424628f6/volumes" Nov 03 21:14:14.151703 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:14:14.151665 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:14:18.922462 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:18.922421 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq"] Nov 03 21:14:18.923052 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:18.922676 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" containerID="cri-o://708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588" gracePeriod=30 Nov 03 21:14:19.855729 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.855646 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh"] Nov 03 21:14:19.857366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857333 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c207bcd6-8186-4416-b183-d806d5ff6f28" containerName="payload-pre-processing" Nov 03 21:14:19.857366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857365 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c207bcd6-8186-4416-b183-d806d5ff6f28" containerName="payload-pre-processing" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857388 5932 cpu_manager.go:414] "containerMap: removing container" podUID="844b4c14-d200-405c-816b-9297609dbe3d" containerName="payload-processing" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857396 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844b4c14-d200-405c-816b-9297609dbe3d" containerName="payload-processing" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857411 5932 cpu_manager.go:414] "containerMap: removing container" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857417 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857426 5932 cpu_manager.go:414] "containerMap: removing container" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857433 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857464 5932 cpu_manager.go:414] "containerMap: removing container" podUID="8cccdaf9-5035-41b5-82e2-237ce524f18c" containerName="revoke-keys" Nov 03 21:14:19.857658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857471 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cccdaf9-5035-41b5-82e2-237ce524f18c" containerName="revoke-keys" Nov 03 21:14:19.858096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857771 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cccdaf9-5035-41b5-82e2-237ce524f18c" containerName="revoke-keys" Nov 03 21:14:19.858096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857804 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c207bcd6-8186-4416-b183-d806d5ff6f28" containerName="payload-pre-processing" Nov 03 21:14:19.858096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857817 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="844b4c14-d200-405c-816b-9297609dbe3d" containerName="payload-processing" Nov 03 21:14:19.858096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857827 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" Nov 03 21:14:19.858096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.857837 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="486253bb-d0a1-4d28-a087-c132a05238ea" containerName="istio-proxy" Nov 03 21:14:19.861483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.861463 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:19.917891 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.917852 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh"] Nov 03 21:14:19.960057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.960027 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:14:19.960436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.960033 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:14:19.983388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.983356 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:19.983590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.983476 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:19.983712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:19.983587 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gh4\" (UniqueName: \"kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.084716 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.084675 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.084925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.084773 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gh4\" (UniqueName: \"kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.084925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.084913 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.085450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.085430 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.087437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.087414 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.150524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.150497 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gh4\" (UniqueName: \"kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4\") pod \"maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.175741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.175712 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:20.571739 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:20.571664 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh"] Nov 03 21:14:20.575845 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:14:20.575815 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432322dd_419d_4d92_ae14_d02696d404c4.slice/crio-9fe0aded401a1cf3c04d5c2e94c2b499c21bd276852b29c18064799aabcbacfa WatchSource:0}: Error finding container 9fe0aded401a1cf3c04d5c2e94c2b499c21bd276852b29c18064799aabcbacfa: Status 404 returned error can't find the container with id 9fe0aded401a1cf3c04d5c2e94c2b499c21bd276852b29c18064799aabcbacfa Nov 03 21:14:21.239111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:21.239076 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" event={"ID":"432322dd-419d-4d92-ae14-d02696d404c4","Type":"ContainerStarted","Data":"52153a27ad56a90530cfc1f0af602d0631f17d35c27e7cbb276910501df4120c"} Nov 03 21:14:21.239531 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:21.239131 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" event={"ID":"432322dd-419d-4d92-ae14-d02696d404c4","Type":"ContainerStarted","Data":"9fe0aded401a1cf3c04d5c2e94c2b499c21bd276852b29c18064799aabcbacfa"} Nov 03 21:14:21.350560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:21.350490 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" podStartSLOduration=2.350470043 podStartE2EDuration="2.350470043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:14:21.34677024 +0000 UTC m=+2669.412011713" watchObservedRunningTime="2026-11-03 21:14:21.350470043 +0000 UTC m=+2669.415711516" Nov 03 21:14:22.248302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:22.248268 5932 generic.go:356] "Generic (PLEG): container finished" podID="432322dd-419d-4d92-ae14-d02696d404c4" containerID="52153a27ad56a90530cfc1f0af602d0631f17d35c27e7cbb276910501df4120c" exitCode=0 Nov 03 21:14:22.248838 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:22.248320 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" event={"ID":"432322dd-419d-4d92-ae14-d02696d404c4","Type":"ContainerDied","Data":"52153a27ad56a90530cfc1f0af602d0631f17d35c27e7cbb276910501df4120c"} Nov 03 21:14:22.307158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:22.307128 5932 patch_prober.go:29] interesting pod/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:14:22.307310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:22.307186 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:14:23.458016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.457991 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:23.530092 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.530026 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca\") pod \"432322dd-419d-4d92-ae14-d02696d404c4\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " Nov 03 21:14:23.530227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.530111 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca\") pod \"432322dd-419d-4d92-ae14-d02696d404c4\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " Nov 03 21:14:23.530273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.530226 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4\" (UniqueName: \"kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4\") pod \"432322dd-419d-4d92-ae14-d02696d404c4\" (UID: \"432322dd-419d-4d92-ae14-d02696d404c4\") " Nov 03 21:14:23.530419 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.530392 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca" pod "432322dd-419d-4d92-ae14-d02696d404c4" (UID: "432322dd-419d-4d92-ae14-d02696d404c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:23.530793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.530772 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:23.532268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.532239 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca" pod "432322dd-419d-4d92-ae14-d02696d404c4" (UID: "432322dd-419d-4d92-ae14-d02696d404c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:14:23.532366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.532247 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4" pod "432322dd-419d-4d92-ae14-d02696d404c4" (UID: "432322dd-419d-4d92-ae14-d02696d404c4"). InnerVolumeSpecName "kube-api-access-w2gh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:23.631898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.631866 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/432322dd-419d-4d92-ae14-d02696d404c4-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:23.631898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:23.631895 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2gh4\" (UniqueName: \"kubernetes.io/projected/432322dd-419d-4d92-ae14-d02696d404c4-kube-api-access-w2gh4\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.171032 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.171008 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:14:24.265277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.265211 5932 generic.go:356] "Generic (PLEG): container finished" podID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerID="708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588" exitCode=0 Nov 03 21:14:24.265277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.265254 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" Nov 03 21:14:24.265452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.265289 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" event={"ID":"d0e6377a-7d4b-4dde-9d95-9c3b271a095b","Type":"ContainerDied","Data":"708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588"} Nov 03 21:14:24.265452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.265342 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq" event={"ID":"d0e6377a-7d4b-4dde-9d95-9c3b271a095b","Type":"ContainerDied","Data":"aff68b152d7dd00f0cc25e1deb3a5fe7c89e0e9f654bbb7f123a5d2ab8d10a9b"} Nov 03 21:14:24.265452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.265364 5932 scope.go:122] "RemoveContainer" containerID="708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588" Nov 03 21:14:24.266708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.266657 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" Nov 03 21:14:24.266708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.266674 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh" event={"ID":"432322dd-419d-4d92-ae14-d02696d404c4","Type":"ContainerDied","Data":"9fe0aded401a1cf3c04d5c2e94c2b499c21bd276852b29c18064799aabcbacfa"} Nov 03 21:14:24.266801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.266716 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fe0aded401a1cf3c04d5c2e94c2b499c21bd276852b29c18064799aabcbacfa" Nov 03 21:14:24.276264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.276247 5932 scope.go:122] "RemoveContainer" containerID="46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee" Nov 03 21:14:24.287136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.287116 5932 scope.go:122] "RemoveContainer" containerID="708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588" Nov 03 21:14:24.287437 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:14:24.287414 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588\": container with ID starting with 708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588 not found: ID does not exist" containerID="708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588" Nov 03 21:14:24.287527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.287446 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588"} err="failed to get container status \"708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588\": rpc error: code = NotFound desc = could not find container \"708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588\": container with ID starting with 708b7b301d91669f534247709e9fce758953a4ef1d4d2201663fef246eb5b588 not found: ID does not exist" Nov 03 21:14:24.287527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.287473 5932 scope.go:122] "RemoveContainer" containerID="46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee" Nov 03 21:14:24.287796 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:14:24.287776 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee\": container with ID starting with 46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee not found: ID does not exist" containerID="46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee" Nov 03 21:14:24.287872 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.287809 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee"} err="failed to get container status \"46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee\": rpc error: code = NotFound desc = could not find container \"46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee\": container with ID starting with 46ae3b826cc2c81b028acf16e0f7233278ecee881b0eb24553ef0737786e13ee not found: ID does not exist" Nov 03 21:14:24.339816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.339794 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.339938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.339824 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.339938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.339868 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.339938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.339898 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.339938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.339928 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.340164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340068 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.340164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340107 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.340164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340143 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.340325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340170 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:24.340325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340215 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data\") pod \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\" (UID: \"d0e6377a-7d4b-4dde-9d95-9c3b271a095b\") " Nov 03 21:14:24.340325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340255 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:24.340325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340269 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:24.340545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340528 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:24.340814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340785 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:14:24.340902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340869 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.340902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340889 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.341006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340903 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.341006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.340918 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.342326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.342304 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:14:24.342326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.342319 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:24.342780 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.342757 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "kube-api-access-c9cgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:14:24.343036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.343006 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy" pod "d0e6377a-7d4b-4dde-9d95-9c3b271a095b" (UID: "d0e6377a-7d4b-4dde-9d95-9c3b271a095b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:14:24.442855 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.442827 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.442962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.442860 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.442962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.442875 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.442962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.442890 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9cgj\" (UniqueName: \"kubernetes.io/projected/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-kube-api-access-c9cgj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.442962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.442904 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e6377a-7d4b-4dde-9d95-9c3b271a095b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:14:24.681173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.681136 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq"] Nov 03 21:14:24.688397 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:24.688358 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8afb98-openshift-default-574c957b6f-8gsjq"] Nov 03 21:14:26.051908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:26.051878 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" path="/var/lib/kubelet/pods/d0e6377a-7d4b-4dde-9d95-9c3b271a095b/volumes" Nov 03 21:14:44.152355 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:14:44.152308 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:14:56.151614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:56.151586 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:14:56.164755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:56.164731 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:14:58.054269 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:58.054239 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9"] Nov 03 21:14:58.094728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:14:58.094700 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cb5f5e3-hpzj9"] Nov 03 21:15:00.053309 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.053268 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ff25ee5-3611-440d-a49b-0e4758a71849" path="/var/lib/kubelet/pods/1ff25ee5-3611-440d-a49b-0e4758a71849/volumes" Nov 03 21:15:00.184353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.184321 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8"] Nov 03 21:15:00.184983 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.184966 5932 cpu_manager.go:414] "containerMap: removing container" podUID="432322dd-419d-4d92-ae14-d02696d404c4" containerName="revoke-keys" Nov 03 21:15:00.185066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.184984 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432322dd-419d-4d92-ae14-d02696d404c4" containerName="revoke-keys" Nov 03 21:15:00.185066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.185013 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" Nov 03 21:15:00.185066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.185019 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" Nov 03 21:15:00.185193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.185130 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="432322dd-419d-4d92-ae14-d02696d404c4" containerName="revoke-keys" Nov 03 21:15:00.185193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.185140 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" Nov 03 21:15:00.185193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.185150 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" Nov 03 21:15:00.188812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.188791 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc"] Nov 03 21:15:00.188946 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.188929 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.189393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.189379 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" Nov 03 21:15:00.189393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.189392 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e6377a-7d4b-4dde-9d95-9c3b271a095b" containerName="istio-proxy" Nov 03 21:15:00.192432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.192414 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw"] Nov 03 21:15:00.192549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.192532 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.196401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.196384 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.231606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.231583 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr"] Nov 03 21:15:00.235504 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.235485 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.236475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.236454 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz"] Nov 03 21:15:00.237505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.237484 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:15:00.240975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.240958 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.276158 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.276132 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8"] Nov 03 21:15:00.313595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313490 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsfgd\" (UniqueName: \"kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.313844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313593 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.313844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313691 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.313844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313738 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bckb4\" (UniqueName: \"kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.313844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313787 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.313844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313816 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.314261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313871 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.314261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313915 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcr9w\" (UniqueName: \"kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w\") pod \"maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.314261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.313974 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf8nm\" (UniqueName: \"kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm\") pod \"maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.314261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.314025 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjj8\" (UniqueName: \"kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.359766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.359725 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc"] Nov 03 21:15:00.363094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.363063 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw"] Nov 03 21:15:00.365867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.365846 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr"] Nov 03 21:15:00.368505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.368485 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz"] Nov 03 21:15:00.399434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.399403 5932 scope.go:122] "RemoveContainer" containerID="2edfdd7bbecaad2b99786725d4846ab95d0fbcd1e9678e022aa0f15f281ed7b8" Nov 03 21:15:00.415470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415443 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.415586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415491 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.415586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415550 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.415732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415591 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcr9w\" (UniqueName: \"kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w\") pod \"maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.415732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415666 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf8nm\" (UniqueName: \"kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm\") pod \"maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.415732 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415719 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjj8\" (UniqueName: \"kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.415912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415781 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsfgd\" (UniqueName: \"kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.415912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415860 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.416035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415937 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.416035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.415988 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bckb4\" (UniqueName: \"kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.416256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.416230 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.416338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.416303 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.416772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.416745 5932 scope.go:122] "RemoveContainer" containerID="00418ae2a9e335e08321dd3cc9e94645ae34b7a9f96eba93749ca694dea71da0" Nov 03 21:15:00.416936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.416912 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.417013 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.416952 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.417579 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.417545 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.428323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.428300 5932 scope.go:122] "RemoveContainer" containerID="29068bbfc9f5e6d128aff50f0c99e21446cd9f6bea9ff12d578ccb3bbbbc586e" Nov 03 21:15:00.437504 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.437488 5932 scope.go:122] "RemoveContainer" containerID="3ed4ac63b3c0ec582b8e689c0522e0205abb467c7fef3c1e78ee566f053bedc7" Nov 03 21:15:00.474586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.473947 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcr9w\" (UniqueName: \"kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w\") pod \"maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.474586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.474139 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bckb4\" (UniqueName: \"kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4\") pod \"maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.474586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.474178 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjj8\" (UniqueName: \"kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8\") pod \"maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.505953 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.505926 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsfgd\" (UniqueName: \"kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd\") pod \"maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.506052 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.505926 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf8nm\" (UniqueName: \"kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm\") pod \"maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.514616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.514588 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:00.520409 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.520392 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:00.549421 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.549401 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:00.569289 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.568953 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:00.805190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.805164 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:00.898420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.898373 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc"] Nov 03 21:15:00.902812 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:00.902780 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd11038_30e3_4150_842b_a49d7a8b8588.slice/crio-6ab405dee43d69b2c65c9d924012dce4c51accd3607f3f2f719dc5a618487dae WatchSource:0}: Error finding container 6ab405dee43d69b2c65c9d924012dce4c51accd3607f3f2f719dc5a618487dae: Status 404 returned error can't find the container with id 6ab405dee43d69b2c65c9d924012dce4c51accd3607f3f2f719dc5a618487dae Nov 03 21:15:00.945651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:00.945611 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw"] Nov 03 21:15:00.950065 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:00.950025 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8e33f2_6527_4edb_b519_9b189a7effe7.slice/crio-566db24c802321a1835585a9cd5eedc4719f422c56491e8cd1b1e5b24a681655 WatchSource:0}: Error finding container 566db24c802321a1835585a9cd5eedc4719f422c56491e8cd1b1e5b24a681655: Status 404 returned error can't find the container with id 566db24c802321a1835585a9cd5eedc4719f422c56491e8cd1b1e5b24a681655 Nov 03 21:15:01.004383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.004353 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr"] Nov 03 21:15:01.008331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.008310 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz"] Nov 03 21:15:01.009085 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:01.009063 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d89581_9c7a_40c7_a496_fd95cb8839ff.slice/crio-0782fd3fe8fff916643a65ab71f19b3facc6ba0a18c188dbe16ddd839e8c15ae WatchSource:0}: Error finding container 0782fd3fe8fff916643a65ab71f19b3facc6ba0a18c188dbe16ddd839e8c15ae: Status 404 returned error can't find the container with id 0782fd3fe8fff916643a65ab71f19b3facc6ba0a18c188dbe16ddd839e8c15ae Nov 03 21:15:01.010043 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:01.010021 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcacb1d5_c224_4ca6_8e59_24e5a26478fe.slice/crio-b3c5759d61c254629f9c1834179af4f346f98607d8568a4e1362b4e251b49e82 WatchSource:0}: Error finding container b3c5759d61c254629f9c1834179af4f346f98607d8568a4e1362b4e251b49e82: Status 404 returned error can't find the container with id b3c5759d61c254629f9c1834179af4f346f98607d8568a4e1362b4e251b49e82 Nov 03 21:15:01.180402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.180375 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8"] Nov 03 21:15:01.185499 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:01.185471 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29031d4e_e617_484d_acbd_f60c504ac0a1.slice/crio-c50837848885276f941e6e7eda0daab2ac956481890209e36866206187803530 WatchSource:0}: Error finding container c50837848885276f941e6e7eda0daab2ac956481890209e36866206187803530: Status 404 returned error can't find the container with id c50837848885276f941e6e7eda0daab2ac956481890209e36866206187803530 Nov 03 21:15:01.561385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.561354 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerStarted","Data":"c50837848885276f941e6e7eda0daab2ac956481890209e36866206187803530"} Nov 03 21:15:01.562756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.562728 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerStarted","Data":"b3c5759d61c254629f9c1834179af4f346f98607d8568a4e1362b4e251b49e82"} Nov 03 21:15:01.564276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.564246 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerStarted","Data":"5b104d746d421a7bf46ead9489d9b2278187b16af83877bce0ecef8dfe6759da"} Nov 03 21:15:01.564408 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.564289 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerStarted","Data":"6ab405dee43d69b2c65c9d924012dce4c51accd3607f3f2f719dc5a618487dae"} Nov 03 21:15:01.565832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.565808 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerStarted","Data":"d7391e51158a4dc7e6d98c808934a3426a57440234494e8ba664334206cf074b"} Nov 03 21:15:01.565940 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.565849 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerStarted","Data":"566db24c802321a1835585a9cd5eedc4719f422c56491e8cd1b1e5b24a681655"} Nov 03 21:15:01.567119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.567094 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerStarted","Data":"0782fd3fe8fff916643a65ab71f19b3facc6ba0a18c188dbe16ddd839e8c15ae"} Nov 03 21:15:01.675681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.675615 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" podStartSLOduration=1.675601087 podStartE2EDuration="1.675601087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.674206342 +0000 UTC m=+2709.739447817" watchObservedRunningTime="2026-11-03 21:15:01.675601087 +0000 UTC m=+2709.740842560" Nov 03 21:15:01.784381 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:01.784267 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" podStartSLOduration=1.784253058 podStartE2EDuration="1.784253058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:01.781837469 +0000 UTC m=+2709.847078943" watchObservedRunningTime="2026-11-03 21:15:01.784253058 +0000 UTC m=+2709.849494530" Nov 03 21:15:02.577448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.577366 5932 generic.go:356] "Generic (PLEG): container finished" podID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerID="2b11e75939bae5f0669cd83243dcbd21875ae1602a695c1f890f683a6e934de3" exitCode=60 Nov 03 21:15:02.577888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.577450 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerDied","Data":"2b11e75939bae5f0669cd83243dcbd21875ae1602a695c1f890f683a6e934de3"} Nov 03 21:15:02.577888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.577771 5932 scope.go:122] "RemoveContainer" containerID="2b11e75939bae5f0669cd83243dcbd21875ae1602a695c1f890f683a6e934de3" Nov 03 21:15:02.579398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.579290 5932 generic.go:356] "Generic (PLEG): container finished" podID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerID="4303f24b5afe1c698290f2fd896a14fd437dc0c04a3abfe3b0d315397fe6a33f" exitCode=60 Nov 03 21:15:02.579398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.579381 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerDied","Data":"4303f24b5afe1c698290f2fd896a14fd437dc0c04a3abfe3b0d315397fe6a33f"} Nov 03 21:15:02.579631 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.579602 5932 scope.go:122] "RemoveContainer" containerID="4303f24b5afe1c698290f2fd896a14fd437dc0c04a3abfe3b0d315397fe6a33f" Nov 03 21:15:02.581277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.581163 5932 generic.go:356] "Generic (PLEG): container finished" podID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerID="8af74613c03f85bb17376ea5662d5c5340613aec991684822e104f0e9dbcdd49" exitCode=60 Nov 03 21:15:02.581277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.581244 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerDied","Data":"8af74613c03f85bb17376ea5662d5c5340613aec991684822e104f0e9dbcdd49"} Nov 03 21:15:02.581455 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.581439 5932 scope.go:122] "RemoveContainer" containerID="8af74613c03f85bb17376ea5662d5c5340613aec991684822e104f0e9dbcdd49" Nov 03 21:15:02.583283 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.583262 5932 generic.go:356] "Generic (PLEG): container finished" podID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerID="5b104d746d421a7bf46ead9489d9b2278187b16af83877bce0ecef8dfe6759da" exitCode=60 Nov 03 21:15:02.583359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.583340 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerDied","Data":"5b104d746d421a7bf46ead9489d9b2278187b16af83877bce0ecef8dfe6759da"} Nov 03 21:15:02.583525 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.583511 5932 scope.go:122] "RemoveContainer" containerID="5b104d746d421a7bf46ead9489d9b2278187b16af83877bce0ecef8dfe6759da" Nov 03 21:15:02.585058 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.585032 5932 generic.go:356] "Generic (PLEG): container finished" podID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerID="d7391e51158a4dc7e6d98c808934a3426a57440234494e8ba664334206cf074b" exitCode=60 Nov 03 21:15:02.585147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.585086 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerDied","Data":"d7391e51158a4dc7e6d98c808934a3426a57440234494e8ba664334206cf074b"} Nov 03 21:15:02.585331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:02.585315 5932 scope.go:122] "RemoveContainer" containerID="d7391e51158a4dc7e6d98c808934a3426a57440234494e8ba664334206cf074b" Nov 03 21:15:03.596295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.596207 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerStarted","Data":"043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3"} Nov 03 21:15:03.598260 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.598234 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerStarted","Data":"83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56"} Nov 03 21:15:03.600190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.600166 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerStarted","Data":"96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d"} Nov 03 21:15:03.601925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.601902 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerStarted","Data":"4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec"} Nov 03 21:15:03.603694 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.603673 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerStarted","Data":"39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f"} Nov 03 21:15:03.706937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.706890 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" podStartSLOduration=3.70687588 podStartE2EDuration="3.70687588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.703275156 +0000 UTC m=+2711.768516629" watchObservedRunningTime="2026-11-03 21:15:03.70687588 +0000 UTC m=+2711.772117355" Nov 03 21:15:03.924087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:03.924039 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" podStartSLOduration=3.924025273 podStartE2EDuration="3.924025273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:03.923790952 +0000 UTC m=+2711.989032425" watchObservedRunningTime="2026-11-03 21:15:03.924025273 +0000 UTC m=+2711.989266746" Nov 03 21:15:04.144145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.144039 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" podStartSLOduration=4.144026206 podStartE2EDuration="4.144026206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:04.142098418 +0000 UTC m=+2712.207339890" watchObservedRunningTime="2026-11-03 21:15:04.144026206 +0000 UTC m=+2712.209267679" Nov 03 21:15:04.613404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.613318 5932 generic.go:356] "Generic (PLEG): container finished" podID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerID="4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec" exitCode=60 Nov 03 21:15:04.613404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.613373 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerDied","Data":"4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec"} Nov 03 21:15:04.613936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.613434 5932 scope.go:122] "RemoveContainer" containerID="5b104d746d421a7bf46ead9489d9b2278187b16af83877bce0ecef8dfe6759da" Nov 03 21:15:04.613936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.613722 5932 scope.go:122] "RemoveContainer" containerID="4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec" Nov 03 21:15:04.614085 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:04.614059 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc_odh-ai-gateway-infra(4fd11038-30e3-4150-842b-a49d7a8b8588)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" Nov 03 21:15:04.620081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.620052 5932 generic.go:356] "Generic (PLEG): container finished" podID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerID="39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f" exitCode=60 Nov 03 21:15:04.620228 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.620137 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerDied","Data":"39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f"} Nov 03 21:15:04.620404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.620389 5932 scope.go:122] "RemoveContainer" containerID="39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f" Nov 03 21:15:04.620718 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:04.620695 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw_odh-ai-gateway-infra(bd8e33f2-6527-4edb-b519-9b189a7effe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" Nov 03 21:15:04.622225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.622203 5932 generic.go:356] "Generic (PLEG): container finished" podID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerID="043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3" exitCode=60 Nov 03 21:15:04.622321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.622282 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerDied","Data":"043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3"} Nov 03 21:15:04.622545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.622528 5932 scope.go:122] "RemoveContainer" containerID="043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3" Nov 03 21:15:04.622834 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:04.622810 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr_odh-ai-gateway-infra(26d89581-9c7a-40c7-a496-fd95cb8839ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" Nov 03 21:15:04.624343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.624318 5932 generic.go:356] "Generic (PLEG): container finished" podID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerID="83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56" exitCode=60 Nov 03 21:15:04.624449 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.624392 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerDied","Data":"83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56"} Nov 03 21:15:04.624607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.624578 5932 scope.go:122] "RemoveContainer" containerID="83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56" Nov 03 21:15:04.624875 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:04.624852 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8_odh-ai-gateway-infra(29031d4e-e617-484d-acbd-f60c504ac0a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" Nov 03 21:15:04.626458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.626439 5932 generic.go:356] "Generic (PLEG): container finished" podID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerID="96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d" exitCode=60 Nov 03 21:15:04.626560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.626478 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerDied","Data":"96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d"} Nov 03 21:15:04.626777 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.626761 5932 scope.go:122] "RemoveContainer" containerID="96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d" Nov 03 21:15:04.627001 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:04.626982 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz_odh-ai-gateway-infra(fcacb1d5-c224-4ca6-8e59-24e5a26478fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" Nov 03 21:15:04.632268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.632248 5932 scope.go:122] "RemoveContainer" containerID="d7391e51158a4dc7e6d98c808934a3426a57440234494e8ba664334206cf074b" Nov 03 21:15:04.645485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.645469 5932 scope.go:122] "RemoveContainer" containerID="2b11e75939bae5f0669cd83243dcbd21875ae1602a695c1f890f683a6e934de3" Nov 03 21:15:04.658247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.658231 5932 scope.go:122] "RemoveContainer" containerID="4303f24b5afe1c698290f2fd896a14fd437dc0c04a3abfe3b0d315397fe6a33f" Nov 03 21:15:04.670952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:04.670934 5932 scope.go:122] "RemoveContainer" containerID="8af74613c03f85bb17376ea5662d5c5340613aec991684822e104f0e9dbcdd49" Nov 03 21:15:05.636173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:05.636142 5932 scope.go:122] "RemoveContainer" containerID="043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3" Nov 03 21:15:05.636676 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:05.636419 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr_odh-ai-gateway-infra(26d89581-9c7a-40c7-a496-fd95cb8839ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" Nov 03 21:15:05.637771 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:05.637748 5932 scope.go:122] "RemoveContainer" containerID="83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56" Nov 03 21:15:05.638013 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:05.637991 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8_odh-ai-gateway-infra(29031d4e-e617-484d-acbd-f60c504ac0a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" Nov 03 21:15:05.639196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:05.639179 5932 scope.go:122] "RemoveContainer" containerID="96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d" Nov 03 21:15:05.639399 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:05.639377 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz_odh-ai-gateway-infra(fcacb1d5-c224-4ca6-8e59-24e5a26478fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" Nov 03 21:15:05.640764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:05.640751 5932 scope.go:122] "RemoveContainer" containerID="4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec" Nov 03 21:15:05.640991 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:05.640970 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc_odh-ai-gateway-infra(4fd11038-30e3-4150-842b-a49d7a8b8588)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" Nov 03 21:15:05.642282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:05.642262 5932 scope.go:122] "RemoveContainer" containerID="39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f" Nov 03 21:15:05.642458 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:05.642439 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw_odh-ai-gateway-infra(bd8e33f2-6527-4edb-b519-9b189a7effe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" Nov 03 21:15:14.153034 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:14.152993 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:15:16.044692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:16.044614 5932 scope.go:122] "RemoveContainer" containerID="043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3" Nov 03 21:15:16.728929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:16.728846 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerStarted","Data":"c989931840a020cf40c4b032828c20c5410949a5cb0cd7e0e17557f79bded6ee"} Nov 03 21:15:17.045305 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.045283 5932 scope.go:122] "RemoveContainer" containerID="39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f" Nov 03 21:15:17.045691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.045359 5932 scope.go:122] "RemoveContainer" containerID="96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d" Nov 03 21:15:17.045691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.045467 5932 scope.go:122] "RemoveContainer" containerID="83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56" Nov 03 21:15:17.738820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.738789 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerStarted","Data":"60cedace49cc4331e9ea5eee1cdbfba98a34dfe8f92758753ef6ca70eb37544c"} Nov 03 21:15:17.740725 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.740695 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerStarted","Data":"a6b6dc32e800ef94caac080a712609638fe459ff10ae9128fd15066cebdc4946"} Nov 03 21:15:17.742691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.742664 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerStarted","Data":"8c26bdb936520a07663e859cdfa8ad5afbca4e2fa537db5de48bfa43db698842"} Nov 03 21:15:17.744483 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.744461 5932 generic.go:356] "Generic (PLEG): container finished" podID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerID="c989931840a020cf40c4b032828c20c5410949a5cb0cd7e0e17557f79bded6ee" exitCode=60 Nov 03 21:15:17.744590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.744502 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerDied","Data":"c989931840a020cf40c4b032828c20c5410949a5cb0cd7e0e17557f79bded6ee"} Nov 03 21:15:17.744590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.744547 5932 scope.go:122] "RemoveContainer" containerID="043afd4f5f9ab16e774c64b9766d9bfa3e3d2d622b7d1a6665d1ad663200e2b3" Nov 03 21:15:17.744863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.744841 5932 scope.go:122] "RemoveContainer" containerID="c989931840a020cf40c4b032828c20c5410949a5cb0cd7e0e17557f79bded6ee" Nov 03 21:15:17.745126 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:17.745105 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr_odh-ai-gateway-infra(26d89581-9c7a-40c7-a496-fd95cb8839ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" Nov 03 21:15:17.849814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:17.849777 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr"] Nov 03 21:15:18.044513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.044483 5932 scope.go:122] "RemoveContainer" containerID="4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec" Nov 03 21:15:18.753775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.753697 5932 generic.go:356] "Generic (PLEG): container finished" podID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerID="60cedace49cc4331e9ea5eee1cdbfba98a34dfe8f92758753ef6ca70eb37544c" exitCode=60 Nov 03 21:15:18.754206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.753768 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerDied","Data":"60cedace49cc4331e9ea5eee1cdbfba98a34dfe8f92758753ef6ca70eb37544c"} Nov 03 21:15:18.754206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.753826 5932 scope.go:122] "RemoveContainer" containerID="83fb6a62a68877f3ac04454aa476cdceff5d118c2f9474a96481de8e4ad85d56" Nov 03 21:15:18.754206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.754195 5932 scope.go:122] "RemoveContainer" containerID="60cedace49cc4331e9ea5eee1cdbfba98a34dfe8f92758753ef6ca70eb37544c" Nov 03 21:15:18.754564 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:18.754541 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8_odh-ai-gateway-infra(29031d4e-e617-484d-acbd-f60c504ac0a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" Nov 03 21:15:18.756539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.756518 5932 generic.go:356] "Generic (PLEG): container finished" podID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerID="a6b6dc32e800ef94caac080a712609638fe459ff10ae9128fd15066cebdc4946" exitCode=60 Nov 03 21:15:18.756696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.756589 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerDied","Data":"a6b6dc32e800ef94caac080a712609638fe459ff10ae9128fd15066cebdc4946"} Nov 03 21:15:18.757053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.757037 5932 scope.go:122] "RemoveContainer" containerID="a6b6dc32e800ef94caac080a712609638fe459ff10ae9128fd15066cebdc4946" Nov 03 21:15:18.757346 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:18.757323 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz_odh-ai-gateway-infra(fcacb1d5-c224-4ca6-8e59-24e5a26478fe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" Nov 03 21:15:18.758650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.758630 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerStarted","Data":"2cbfc79a72f19d64b880c9352e281b481b3d74571b6097e4487d8a759f5f4fdd"} Nov 03 21:15:18.760638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.760595 5932 generic.go:356] "Generic (PLEG): container finished" podID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerID="8c26bdb936520a07663e859cdfa8ad5afbca4e2fa537db5de48bfa43db698842" exitCode=60 Nov 03 21:15:18.760731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.760652 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerDied","Data":"8c26bdb936520a07663e859cdfa8ad5afbca4e2fa537db5de48bfa43db698842"} Nov 03 21:15:18.760944 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.760925 5932 scope.go:122] "RemoveContainer" containerID="8c26bdb936520a07663e859cdfa8ad5afbca4e2fa537db5de48bfa43db698842" Nov 03 21:15:18.761148 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:18.761122 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw_odh-ai-gateway-infra(bd8e33f2-6527-4edb-b519-9b189a7effe7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" Nov 03 21:15:18.769927 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.769898 5932 scope.go:122] "RemoveContainer" containerID="96b82a821626d18f0455e51787beadabd8eed371117ee33d4efc78b9f92c143d" Nov 03 21:15:18.783876 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.783849 5932 scope.go:122] "RemoveContainer" containerID="39fd8fbe4eeaa817a7ab2e4d2aefee1a815a8cc0eaf17454fcf60e1e43654d2f" Nov 03 21:15:18.865957 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.865930 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8"] Nov 03 21:15:18.930588 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:18.930558 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:19.065734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.065708 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8\" (UniqueName: \"kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8\") pod \"26d89581-9c7a-40c7-a496-fd95cb8839ff\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " Nov 03 21:15:19.065850 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.065797 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca\") pod \"26d89581-9c7a-40c7-a496-fd95cb8839ff\" (UID: \"26d89581-9c7a-40c7-a496-fd95cb8839ff\") " Nov 03 21:15:19.066109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.066082 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca" pod "26d89581-9c7a-40c7-a496-fd95cb8839ff" (UID: "26d89581-9c7a-40c7-a496-fd95cb8839ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:19.067904 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.067883 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8" pod "26d89581-9c7a-40c7-a496-fd95cb8839ff" (UID: "26d89581-9c7a-40c7-a496-fd95cb8839ff"). InnerVolumeSpecName "kube-api-access-mmjj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:19.077817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.077795 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw"] Nov 03 21:15:19.167144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.167111 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26d89581-9c7a-40c7-a496-fd95cb8839ff-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:19.167144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.167144 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjj8\" (UniqueName: \"kubernetes.io/projected/26d89581-9c7a-40c7-a496-fd95cb8839ff-kube-api-access-mmjj8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:19.183933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.183902 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz"] Nov 03 21:15:19.462045 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.462012 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx"] Nov 03 21:15:19.462235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.462211 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" containerID="cri-o://36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b" gracePeriod=30 Nov 03 21:15:19.776979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.776880 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" Nov 03 21:15:19.776979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.776911 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr" event={"ID":"26d89581-9c7a-40c7-a496-fd95cb8839ff","Type":"ContainerDied","Data":"0782fd3fe8fff916643a65ab71f19b3facc6ba0a18c188dbe16ddd839e8c15ae"} Nov 03 21:15:19.777491 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.776988 5932 scope.go:122] "RemoveContainer" containerID="c989931840a020cf40c4b032828c20c5410949a5cb0cd7e0e17557f79bded6ee" Nov 03 21:15:19.784004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.783977 5932 generic.go:356] "Generic (PLEG): container finished" podID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerID="2cbfc79a72f19d64b880c9352e281b481b3d74571b6097e4487d8a759f5f4fdd" exitCode=60 Nov 03 21:15:19.784117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.784049 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerDied","Data":"2cbfc79a72f19d64b880c9352e281b481b3d74571b6097e4487d8a759f5f4fdd"} Nov 03 21:15:19.784463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.784442 5932 scope.go:122] "RemoveContainer" containerID="2cbfc79a72f19d64b880c9352e281b481b3d74571b6097e4487d8a759f5f4fdd" Nov 03 21:15:19.784766 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:19.784745 5932 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc_odh-ai-gateway-infra(4fd11038-30e3-4150-842b-a49d7a8b8588)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" Nov 03 21:15:19.793190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.793165 5932 scope.go:122] "RemoveContainer" containerID="4b68fa1699b7e0c3321d9ef855b1d4470920343f58224a6830ef843b4c7396ec" Nov 03 21:15:19.959216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:19.959186 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:20.027521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.027461 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:20.030767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.030747 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:20.086769 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.086740 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca\") pod \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " Nov 03 21:15:20.086992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.086976 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm\" (UniqueName: \"kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm\") pod \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\" (UID: \"fcacb1d5-c224-4ca6-8e59-24e5a26478fe\") " Nov 03 21:15:20.087102 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.087081 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca" pod "fcacb1d5-c224-4ca6-8e59-24e5a26478fe" (UID: "fcacb1d5-c224-4ca6-8e59-24e5a26478fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:20.087350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.087332 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.088932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.088906 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm" pod "fcacb1d5-c224-4ca6-8e59-24e5a26478fe" (UID: "fcacb1d5-c224-4ca6-8e59-24e5a26478fe"). InnerVolumeSpecName "kube-api-access-tf8nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:20.153394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.153358 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr"] Nov 03 21:15:20.157443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.157417 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-88228a-29895675-smdjr"] Nov 03 21:15:20.188672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.188651 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4\" (UniqueName: \"kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4\") pod \"bd8e33f2-6527-4edb-b519-9b189a7effe7\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " Nov 03 21:15:20.188849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.188835 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca\") pod \"bd8e33f2-6527-4edb-b519-9b189a7effe7\" (UID: \"bd8e33f2-6527-4edb-b519-9b189a7effe7\") " Nov 03 21:15:20.188909 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.188895 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd\" (UniqueName: \"kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd\") pod \"29031d4e-e617-484d-acbd-f60c504ac0a1\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " Nov 03 21:15:20.188973 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.188938 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca\") pod \"29031d4e-e617-484d-acbd-f60c504ac0a1\" (UID: \"29031d4e-e617-484d-acbd-f60c504ac0a1\") " Nov 03 21:15:20.189351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.189322 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca" pod "29031d4e-e617-484d-acbd-f60c504ac0a1" (UID: "29031d4e-e617-484d-acbd-f60c504ac0a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:20.189555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.189329 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca" pod "bd8e33f2-6527-4edb-b519-9b189a7effe7" (UID: "bd8e33f2-6527-4edb-b519-9b189a7effe7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:20.189719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.189653 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29031d4e-e617-484d-acbd-f60c504ac0a1-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.189719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.189677 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf8nm\" (UniqueName: \"kubernetes.io/projected/fcacb1d5-c224-4ca6-8e59-24e5a26478fe-kube-api-access-tf8nm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.191069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.191049 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4" pod "bd8e33f2-6527-4edb-b519-9b189a7effe7" (UID: "bd8e33f2-6527-4edb-b519-9b189a7effe7"). InnerVolumeSpecName "kube-api-access-bckb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:20.191197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.191179 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd" pod "29031d4e-e617-484d-acbd-f60c504ac0a1" (UID: "29031d4e-e617-484d-acbd-f60c504ac0a1"). InnerVolumeSpecName "kube-api-access-xsfgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:20.291286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.291221 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8e33f2-6527-4edb-b519-9b189a7effe7-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.291286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.291243 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsfgd\" (UniqueName: \"kubernetes.io/projected/29031d4e-e617-484d-acbd-f60c504ac0a1-kube-api-access-xsfgd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.291286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.291254 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bckb4\" (UniqueName: \"kubernetes.io/projected/bd8e33f2-6527-4edb-b519-9b189a7effe7-kube-api-access-bckb4\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:20.336990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.336959 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc"] Nov 03 21:15:20.797098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.797065 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" Nov 03 21:15:20.797556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.797069 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8" event={"ID":"29031d4e-e617-484d-acbd-f60c504ac0a1","Type":"ContainerDied","Data":"c50837848885276f941e6e7eda0daab2ac956481890209e36866206187803530"} Nov 03 21:15:20.797556 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.797180 5932 scope.go:122] "RemoveContainer" containerID="60cedace49cc4331e9ea5eee1cdbfba98a34dfe8f92758753ef6ca70eb37544c" Nov 03 21:15:20.798755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.798732 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" Nov 03 21:15:20.798874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.798756 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz" event={"ID":"fcacb1d5-c224-4ca6-8e59-24e5a26478fe","Type":"ContainerDied","Data":"b3c5759d61c254629f9c1834179af4f346f98607d8568a4e1362b4e251b49e82"} Nov 03 21:15:20.803137 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.803112 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" Nov 03 21:15:20.803137 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.803122 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw" event={"ID":"bd8e33f2-6527-4edb-b519-9b189a7effe7","Type":"ContainerDied","Data":"566db24c802321a1835585a9cd5eedc4719f422c56491e8cd1b1e5b24a681655"} Nov 03 21:15:20.811155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.811063 5932 scope.go:122] "RemoveContainer" containerID="a6b6dc32e800ef94caac080a712609638fe459ff10ae9128fd15066cebdc4946" Nov 03 21:15:20.821203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.821182 5932 scope.go:122] "RemoveContainer" containerID="8c26bdb936520a07663e859cdfa8ad5afbca4e2fa537db5de48bfa43db698842" Nov 03 21:15:20.955410 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.955374 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8"] Nov 03 21:15:20.960402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.960376 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-504eb0-29895675-9dsw8"] Nov 03 21:15:20.969458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:20.969438 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:21.025478 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.025435 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w\" (UniqueName: \"kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w\") pod \"4fd11038-30e3-4150-842b-a49d7a8b8588\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " Nov 03 21:15:21.032447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.032410 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca\") pod \"4fd11038-30e3-4150-842b-a49d7a8b8588\" (UID: \"4fd11038-30e3-4150-842b-a49d7a8b8588\") " Nov 03 21:15:21.042535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.038034 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca" pod "4fd11038-30e3-4150-842b-a49d7a8b8588" (UID: "4fd11038-30e3-4150-842b-a49d7a8b8588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:21.043326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.043299 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fd11038-30e3-4150-842b-a49d7a8b8588-maas-api-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:21.044857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.044824 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w" pod "4fd11038-30e3-4150-842b-a49d7a8b8588" (UID: "4fd11038-30e3-4150-842b-a49d7a8b8588"). InnerVolumeSpecName "kube-api-access-rcr9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:21.070135 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.070111 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw"] Nov 03 21:15:21.076258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.076237 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0cd49b-29895675-bnzsw"] Nov 03 21:15:21.144666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.144635 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcr9w\" (UniqueName: \"kubernetes.io/projected/4fd11038-30e3-4150-842b-a49d7a8b8588-kube-api-access-rcr9w\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:21.180417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.180382 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz"] Nov 03 21:15:21.183833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.183810 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-10fa59-29895675-587zz"] Nov 03 21:15:21.814372 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.814341 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" Nov 03 21:15:21.814865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.814347 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc" event={"ID":"4fd11038-30e3-4150-842b-a49d7a8b8588","Type":"ContainerDied","Data":"6ab405dee43d69b2c65c9d924012dce4c51accd3607f3f2f719dc5a618487dae"} Nov 03 21:15:21.814865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.814537 5932 scope.go:122] "RemoveContainer" containerID="2cbfc79a72f19d64b880c9352e281b481b3d74571b6097e4487d8a759f5f4fdd" Nov 03 21:15:21.969275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.969244 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc"] Nov 03 21:15:21.972659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:21.972606 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fc4e8d-29895675-867sc"] Nov 03 21:15:22.053024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:22.052996 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" path="/var/lib/kubelet/pods/26d89581-9c7a-40c7-a496-fd95cb8839ff/volumes" Nov 03 21:15:22.053428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:22.053414 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" path="/var/lib/kubelet/pods/29031d4e-e617-484d-acbd-f60c504ac0a1/volumes" Nov 03 21:15:22.053870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:22.053855 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" path="/var/lib/kubelet/pods/4fd11038-30e3-4150-842b-a49d7a8b8588/volumes" Nov 03 21:15:22.054540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:22.054525 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" path="/var/lib/kubelet/pods/bd8e33f2-6527-4edb-b519-9b189a7effe7/volumes" Nov 03 21:15:22.054961 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:22.054947 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" path="/var/lib/kubelet/pods/fcacb1d5-c224-4ca6-8e59-24e5a26478fe/volumes" Nov 03 21:15:23.274843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.274794 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h"] Nov 03 21:15:23.275385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.275077 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" containerID="cri-o://d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa" gracePeriod=30 Nov 03 21:15:23.420545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.420514 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc"] Nov 03 21:15:23.421202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421182 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:23.421202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421201 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421214 5932 cpu_manager.go:414] "containerMap: removing container" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421219 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421229 5932 cpu_manager.go:414] "containerMap: removing container" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421235 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421250 5932 cpu_manager.go:414] "containerMap: removing container" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421255 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421264 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421270 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421280 5932 cpu_manager.go:414] "containerMap: removing container" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421286 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421301 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421305 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421311 5932 cpu_manager.go:414] "containerMap: removing container" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421315 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421322 5932 cpu_manager.go:414] "containerMap: removing container" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421326 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421333 5932 cpu_manager.go:414] "containerMap: removing container" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421338 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421346 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421350 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421365 5932 cpu_manager.go:414] "containerMap: removing container" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:23.421365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421369 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421452 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421461 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421468 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421474 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421480 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421484 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421490 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421495 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421502 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421509 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421515 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.422097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.421521 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="29031d4e-e617-484d-acbd-f60c504ac0a1" containerName="cleanup" Nov 03 21:15:23.426705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.426684 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.475695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.475667 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:15:23.475695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.475682 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:15:23.478488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.478464 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc"] Nov 03 21:15:23.490668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.490607 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.490798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.490754 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89qd\" (UniqueName: \"kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.490931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.490915 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.592788 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.592706 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.592960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.592847 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.592960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.592934 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89qd\" (UniqueName: \"kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.593613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.593585 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.595268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.595247 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.651190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.651157 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89qd\" (UniqueName: \"kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd\") pod \"maas-api-revoke-sub-0dba083a-bhvsc\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:23.741742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:23.741714 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:24.115412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.115376 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc"] Nov 03 21:15:24.120640 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:24.120600 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf654c0fc_d13f_49a9_aee6_7470756118a2.slice/crio-d08810f07c1c4e692cea482280f6b3dc25fed52e1a7e8f61ee6dabcaf7d150ff WatchSource:0}: Error finding container d08810f07c1c4e692cea482280f6b3dc25fed52e1a7e8f61ee6dabcaf7d150ff: Status 404 returned error can't find the container with id d08810f07c1c4e692cea482280f6b3dc25fed52e1a7e8f61ee6dabcaf7d150ff Nov 03 21:15:24.737885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.737861 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 21:15:24.808893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.808825 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.808905 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.808944 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809040 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809075 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809116 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:24.809164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809153 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809185 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809221 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809275 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert\") pod \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\" (UID: \"c4826cc8-9dd1-4e79-a2ba-8df64125e818\") " Nov 03 21:15:24.809370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809339 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:24.809370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809350 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:24.809669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809569 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:24.809773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809750 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:24.809950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809932 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.810022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809952 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.810022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809961 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.810022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809970 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.810022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.809979 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.811460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.811437 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "kube-api-access-8f28k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:24.811460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.811448 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:24.811724 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.811701 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:24.811782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.811742 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy" pod "c4826cc8-9dd1-4e79-a2ba-8df64125e818" (UID: "c4826cc8-9dd1-4e79-a2ba-8df64125e818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.853942 5932 generic.go:356] "Generic (PLEG): container finished" podID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerID="36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b" exitCode=0 Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.854023 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.854155 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" event={"ID":"c4826cc8-9dd1-4e79-a2ba-8df64125e818","Type":"ContainerDied","Data":"36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b"} Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.854286 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx" event={"ID":"c4826cc8-9dd1-4e79-a2ba-8df64125e818","Type":"ContainerDied","Data":"59bb2ff98340e0ea65b0504c09a23cf98eacaecd30ff0d31cc0a55b2853b3978"} Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.854324 5932 scope.go:122] "RemoveContainer" containerID="36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b" Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.857529 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" event={"ID":"f654c0fc-d13f-49a9-aee6-7470756118a2","Type":"ContainerStarted","Data":"866b75dbb4cc1987de2daa0595af0c26bc31c30bf9bb3b60505c3c6a6c86f2b3"} Nov 03 21:15:24.858822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.857579 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" event={"ID":"f654c0fc-d13f-49a9-aee6-7470756118a2","Type":"ContainerStarted","Data":"d08810f07c1c4e692cea482280f6b3dc25fed52e1a7e8f61ee6dabcaf7d150ff"} Nov 03 21:15:24.878578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.872576 5932 scope.go:122] "RemoveContainer" containerID="1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892" Nov 03 21:15:24.891453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.891430 5932 scope.go:122] "RemoveContainer" containerID="36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b" Nov 03 21:15:24.891833 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:24.891801 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b\": container with ID starting with 36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b not found: ID does not exist" containerID="36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b" Nov 03 21:15:24.891917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.891844 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b"} err="failed to get container status \"36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b\": rpc error: code = NotFound desc = could not find container \"36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b\": container with ID starting with 36f695561c842852f1692adf92faaaac27e7ee62576be49c5c011dc3da52c65b not found: ID does not exist" Nov 03 21:15:24.891917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.891869 5932 scope.go:122] "RemoveContainer" containerID="1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892" Nov 03 21:15:24.892188 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:24.892164 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892\": container with ID starting with 1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892 not found: ID does not exist" containerID="1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892" Nov 03 21:15:24.892245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.892197 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892"} err="failed to get container status \"1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892\": rpc error: code = NotFound desc = could not find container \"1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892\": container with ID starting with 1ceb9392fc6611a283891a3e6f93649d267ff1c541d5c6f617955bc202ec9892 not found: ID does not exist" Nov 03 21:15:24.911529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.911504 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f28k\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-kube-api-access-8f28k\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.911645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.911533 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.911645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.911548 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:24.911645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:24.911563 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4826cc8-9dd1-4e79-a2ba-8df64125e818-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:25.010595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:25.010558 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx"] Nov 03 21:15:25.013949 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:25.013926 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3b8feb-openshift-default-97d469769-h4ttx"] Nov 03 21:15:25.119469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:25.119378 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" podStartSLOduration=2.119365093 podStartE2EDuration="2.119365093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:25.117722663 +0000 UTC m=+2733.182964136" watchObservedRunningTime="2026-11-03 21:15:25.119365093 +0000 UTC m=+2733.184606566" Nov 03 21:15:25.869202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:25.869120 5932 generic.go:356] "Generic (PLEG): container finished" podID="f654c0fc-d13f-49a9-aee6-7470756118a2" containerID="866b75dbb4cc1987de2daa0595af0c26bc31c30bf9bb3b60505c3c6a6c86f2b3" exitCode=0 Nov 03 21:15:25.869578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:25.869200 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" event={"ID":"f654c0fc-d13f-49a9-aee6-7470756118a2","Type":"ContainerDied","Data":"866b75dbb4cc1987de2daa0595af0c26bc31c30bf9bb3b60505c3c6a6c86f2b3"} Nov 03 21:15:26.051760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:26.051727 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" path="/var/lib/kubelet/pods/c4826cc8-9dd1-4e79-a2ba-8df64125e818/volumes" Nov 03 21:15:27.027834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.027806 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:27.138569 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.138492 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca\") pod \"f654c0fc-d13f-49a9-aee6-7470756118a2\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " Nov 03 21:15:27.138569 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.138559 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd\" (UniqueName: \"kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd\") pod \"f654c0fc-d13f-49a9-aee6-7470756118a2\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " Nov 03 21:15:27.138810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.138781 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca\") pod \"f654c0fc-d13f-49a9-aee6-7470756118a2\" (UID: \"f654c0fc-d13f-49a9-aee6-7470756118a2\") " Nov 03 21:15:27.138986 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.138966 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca" pod "f654c0fc-d13f-49a9-aee6-7470756118a2" (UID: "f654c0fc-d13f-49a9-aee6-7470756118a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:27.139315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.139291 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:27.140886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.140861 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca" pod "f654c0fc-d13f-49a9-aee6-7470756118a2" (UID: "f654c0fc-d13f-49a9-aee6-7470756118a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:27.140886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.140872 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd" pod "f654c0fc-d13f-49a9-aee6-7470756118a2" (UID: "f654c0fc-d13f-49a9-aee6-7470756118a2"). InnerVolumeSpecName "kube-api-access-n89qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:27.240229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.240201 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f654c0fc-d13f-49a9-aee6-7470756118a2-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:27.240229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.240227 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89qd\" (UniqueName: \"kubernetes.io/projected/f654c0fc-d13f-49a9-aee6-7470756118a2-kube-api-access-n89qd\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:27.885679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.885652 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" Nov 03 21:15:27.885679 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.885666 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc" event={"ID":"f654c0fc-d13f-49a9-aee6-7470756118a2","Type":"ContainerDied","Data":"d08810f07c1c4e692cea482280f6b3dc25fed52e1a7e8f61ee6dabcaf7d150ff"} Nov 03 21:15:27.886085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:27.885710 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d08810f07c1c4e692cea482280f6b3dc25fed52e1a7e8f61ee6dabcaf7d150ff" Nov 03 21:15:28.548816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.548744 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 21:15:28.653357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653327 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.653522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653418 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.653522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653467 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.653677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653521 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.653677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653557 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.653677 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653657 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.653899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653722 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:28.653899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.653783 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.654053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654010 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:28.654053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654034 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:28.654162 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654104 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.654225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654208 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data\") pod \"6f9a630a-9407-40e4-b964-61b39dd48e75\" (UID: \"6f9a630a-9407-40e4-b964-61b39dd48e75\") " Nov 03 21:15:28.654501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654473 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:28.654676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654591 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:28.654798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654780 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.654868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654800 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.654868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654810 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9a630a-9407-40e4-b964-61b39dd48e75-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.654868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654819 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.654868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.654828 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.655841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.655821 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:28.655966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.655932 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:28.656069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.656017 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:28.656180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.656160 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2" pod "6f9a630a-9407-40e4-b964-61b39dd48e75" (UID: "6f9a630a-9407-40e4-b964-61b39dd48e75"). InnerVolumeSpecName "kube-api-access-j9qz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:28.755680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.755616 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qz2\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-kube-api-access-j9qz2\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.755680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.755676 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.755680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.755687 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.756007 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.755696 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9a630a-9407-40e4-b964-61b39dd48e75-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:28.894335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.894305 5932 generic.go:356] "Generic (PLEG): container finished" podID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerID="d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa" exitCode=0 Nov 03 21:15:28.894522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.894390 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" event={"ID":"6f9a630a-9407-40e4-b964-61b39dd48e75","Type":"ContainerDied","Data":"d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa"} Nov 03 21:15:28.899544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.894686 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" event={"ID":"6f9a630a-9407-40e4-b964-61b39dd48e75","Type":"ContainerDied","Data":"ea9258e14fc17a24572076c7457d67e2730673306a03866ef13d540ed8802a5f"} Nov 03 21:15:28.899544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.894878 5932 scope.go:122] "RemoveContainer" containerID="d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa" Nov 03 21:15:28.899544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.894923 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h" Nov 03 21:15:28.915578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.915556 5932 scope.go:122] "RemoveContainer" containerID="60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03" Nov 03 21:15:28.925114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.925091 5932 scope.go:122] "RemoveContainer" containerID="d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa" Nov 03 21:15:28.925410 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:28.925390 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa\": container with ID starting with d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa not found: ID does not exist" containerID="d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa" Nov 03 21:15:28.925477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.925422 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa"} err="failed to get container status \"d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa\": rpc error: code = NotFound desc = could not find container \"d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa\": container with ID starting with d31d2835ea1ef7097ad3d6fb49d353b6cd97202bf0d5297b3d6bfbf053dd15fa not found: ID does not exist" Nov 03 21:15:28.925477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.925441 5932 scope.go:122] "RemoveContainer" containerID="60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03" Nov 03 21:15:28.925698 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:28.925683 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03\": container with ID starting with 60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03 not found: ID does not exist" containerID="60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03" Nov 03 21:15:28.925745 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:28.925700 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03"} err="failed to get container status \"60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03\": rpc error: code = NotFound desc = could not find container \"60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03\": container with ID starting with 60401e69c3524bcf36e66caf4b71afab6f1bb0906778f07c5772a25ad9139d03 not found: ID does not exist" Nov 03 21:15:29.059699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:29.059670 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h"] Nov 03 21:15:29.063812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:29.063775 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0cd49b-openshift-default-7759df4c47-nxf8h"] Nov 03 21:15:30.051762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:30.051729 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" path="/var/lib/kubelet/pods/6f9a630a-9407-40e4-b964-61b39dd48e75/volumes" Nov 03 21:15:37.871026 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.870986 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc"] Nov 03 21:15:37.871778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871757 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" Nov 03 21:15:37.871778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871778 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871795 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871802 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871816 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871823 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871833 5932 cpu_manager.go:414] "containerMap: removing container" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871839 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871893 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f654c0fc-d13f-49a9-aee6-7470756118a2" containerName="revoke-keys" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871899 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f654c0fc-d13f-49a9-aee6-7470756118a2" containerName="revoke-keys" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871915 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871922 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcacb1d5-c224-4ca6-8e59-24e5a26478fe" containerName="cleanup" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871947 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871954 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" Nov 03 21:15:37.871968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871972 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.871979 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872133 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872150 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4826cc8-9dd1-4e79-a2ba-8df64125e818" containerName="istio-proxy" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872160 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8e33f2-6527-4edb-b519-9b189a7effe7" containerName="cleanup" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872178 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f654c0fc-d13f-49a9-aee6-7470756118a2" containerName="revoke-keys" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872191 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="26d89581-9c7a-40c7-a496-fd95cb8839ff" containerName="cleanup" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872199 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd11038-30e3-4150-842b-a49d7a8b8588" containerName="cleanup" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872211 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" Nov 03 21:15:37.872645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.872221 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9a630a-9407-40e4-b964-61b39dd48e75" containerName="istio-proxy" Nov 03 21:15:37.877598 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.877577 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:37.972741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.972711 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:15:37.972741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.972720 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:15:37.976064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.976041 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc"] Nov 03 21:15:37.976414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.976390 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvksf\" (UniqueName: \"kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:37.976452 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.976428 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:37.976591 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:37.976577 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.078119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.078089 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.078290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.078236 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvksf\" (UniqueName: \"kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.078290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.078269 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.078921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.078900 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.080800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.080777 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.136043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.136014 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvksf\" (UniqueName: \"kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf\") pod \"maas-api-revoke-sub-42b23bde-qg5dc\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.191798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.191770 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:38.576516 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.576413 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc"] Nov 03 21:15:38.581929 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:38.581899 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a37e9d4_d5db_4538_8b63_53e1da441f51.slice/crio-caa89ae5be5d7c2b4c9fc978d7002123b23ce88f8f8213897a274224d879b256 WatchSource:0}: Error finding container caa89ae5be5d7c2b4c9fc978d7002123b23ce88f8f8213897a274224d879b256: Status 404 returned error can't find the container with id caa89ae5be5d7c2b4c9fc978d7002123b23ce88f8f8213897a274224d879b256 Nov 03 21:15:38.993636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.993594 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" event={"ID":"9a37e9d4-d5db-4538-8b63-53e1da441f51","Type":"ContainerStarted","Data":"23e717196b8d76f05ff41c2dbc3bfdd58c3f91277f9ddb85aa81cfcd56311d2d"} Nov 03 21:15:38.993992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:38.993678 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" event={"ID":"9a37e9d4-d5db-4538-8b63-53e1da441f51","Type":"ContainerStarted","Data":"caa89ae5be5d7c2b4c9fc978d7002123b23ce88f8f8213897a274224d879b256"} Nov 03 21:15:39.112696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:39.108409 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" podStartSLOduration=2.108390535 podStartE2EDuration="2.108390535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:39.102425224 +0000 UTC m=+2747.167666698" watchObservedRunningTime="2026-11-03 21:15:39.108390535 +0000 UTC m=+2747.173632009" Nov 03 21:15:40.002933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:40.002898 5932 generic.go:356] "Generic (PLEG): container finished" podID="9a37e9d4-d5db-4538-8b63-53e1da441f51" containerID="23e717196b8d76f05ff41c2dbc3bfdd58c3f91277f9ddb85aa81cfcd56311d2d" exitCode=0 Nov 03 21:15:40.003356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:40.002968 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" event={"ID":"9a37e9d4-d5db-4538-8b63-53e1da441f51","Type":"ContainerDied","Data":"23e717196b8d76f05ff41c2dbc3bfdd58c3f91277f9ddb85aa81cfcd56311d2d"} Nov 03 21:15:41.161464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.161440 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:41.213382 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.213350 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca\") pod \"9a37e9d4-d5db-4538-8b63-53e1da441f51\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " Nov 03 21:15:41.213534 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.213461 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca\") pod \"9a37e9d4-d5db-4538-8b63-53e1da441f51\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " Nov 03 21:15:41.213534 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.213526 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf\" (UniqueName: \"kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf\") pod \"9a37e9d4-d5db-4538-8b63-53e1da441f51\" (UID: \"9a37e9d4-d5db-4538-8b63-53e1da441f51\") " Nov 03 21:15:41.213680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.213610 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca" pod "9a37e9d4-d5db-4538-8b63-53e1da441f51" (UID: "9a37e9d4-d5db-4538-8b63-53e1da441f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:41.214024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.214004 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.215530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.215502 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca" pod "9a37e9d4-d5db-4538-8b63-53e1da441f51" (UID: "9a37e9d4-d5db-4538-8b63-53e1da441f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:41.215667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.215601 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf" pod "9a37e9d4-d5db-4538-8b63-53e1da441f51" (UID: "9a37e9d4-d5db-4538-8b63-53e1da441f51"). InnerVolumeSpecName "kube-api-access-gvksf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:41.315669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.315581 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a37e9d4-d5db-4538-8b63-53e1da441f51-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:41.315669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:41.315610 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvksf\" (UniqueName: \"kubernetes.io/projected/9a37e9d4-d5db-4538-8b63-53e1da441f51-kube-api-access-gvksf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:42.019790 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:42.019762 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" Nov 03 21:15:42.019974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:42.019804 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc" event={"ID":"9a37e9d4-d5db-4538-8b63-53e1da441f51","Type":"ContainerDied","Data":"caa89ae5be5d7c2b4c9fc978d7002123b23ce88f8f8213897a274224d879b256"} Nov 03 21:15:42.019974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:42.019857 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caa89ae5be5d7c2b4c9fc978d7002123b23ce88f8f8213897a274224d879b256" Nov 03 21:15:43.604003 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:43.603971 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk"] Nov 03 21:15:43.650006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:43.649976 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-10fa59-gfnpk"] Nov 03 21:15:44.053330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:44.053297 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cccdaf9-5035-41b5-82e2-237ce524f18c" path="/var/lib/kubelet/pods/8cccdaf9-5035-41b5-82e2-237ce524f18c/volumes" Nov 03 21:15:44.245671 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:15:44.243153 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:15:50.900280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:50.900243 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn"] Nov 03 21:15:50.900788 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:50.900607 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" containerID="cri-o://fa355741cd1f79cc395b04ccccf7baa74ce38fe4c8ea5b0dc3db1a9ee563d7c2" gracePeriod=30 Nov 03 21:15:52.870018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.869981 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx"] Nov 03 21:15:52.870572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.870556 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9a37e9d4-d5db-4538-8b63-53e1da441f51" containerName="revoke-keys" Nov 03 21:15:52.870572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.870571 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a37e9d4-d5db-4538-8b63-53e1da441f51" containerName="revoke-keys" Nov 03 21:15:52.870716 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.870705 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a37e9d4-d5db-4538-8b63-53e1da441f51" containerName="revoke-keys" Nov 03 21:15:52.874339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.874318 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:52.928098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.928069 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx"] Nov 03 21:15:52.969119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.969086 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:15:52.970743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.970715 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:15:52.979935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.979900 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:52.980110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.980092 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:52.980168 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:52.980154 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6gx\" (UniqueName: \"kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.081186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.081146 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.081402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.081203 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6gx\" (UniqueName: \"kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.081402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.081272 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.082016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.081992 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.083784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.083742 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.138380 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.138348 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6gx\" (UniqueName: \"kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx\") pod \"maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.188557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.188531 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:53.576059 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:53.575976 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx"] Nov 03 21:15:53.580584 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:15:53.580557 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260b4425_00a8_421f_8b53_b41a2382ab86.slice/crio-4f0e2d4c713abb03834d4f3a1c0d67c54ad8f16b8f62932db172c623ed851320 WatchSource:0}: Error finding container 4f0e2d4c713abb03834d4f3a1c0d67c54ad8f16b8f62932db172c623ed851320: Status 404 returned error can't find the container with id 4f0e2d4c713abb03834d4f3a1c0d67c54ad8f16b8f62932db172c623ed851320 Nov 03 21:15:54.108767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:54.108730 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" event={"ID":"260b4425-00a8-421f-8b53-b41a2382ab86","Type":"ContainerStarted","Data":"11dd445d26d9b3face72c5bc8d50ebb3db6ba80b3268d0dacdfe9fac9ac99709"} Nov 03 21:15:54.109152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:54.108800 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" event={"ID":"260b4425-00a8-421f-8b53-b41a2382ab86","Type":"ContainerStarted","Data":"4f0e2d4c713abb03834d4f3a1c0d67c54ad8f16b8f62932db172c623ed851320"} Nov 03 21:15:54.219436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:54.219338 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" podStartSLOduration=2.219323435 podStartE2EDuration="2.219323435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:15:54.216348521 +0000 UTC m=+2762.281589996" watchObservedRunningTime="2026-11-03 21:15:54.219323435 +0000 UTC m=+2762.284564905" Nov 03 21:15:55.118334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:55.118291 5932 generic.go:356] "Generic (PLEG): container finished" podID="260b4425-00a8-421f-8b53-b41a2382ab86" containerID="11dd445d26d9b3face72c5bc8d50ebb3db6ba80b3268d0dacdfe9fac9ac99709" exitCode=0 Nov 03 21:15:55.118765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:55.118375 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" event={"ID":"260b4425-00a8-421f-8b53-b41a2382ab86","Type":"ContainerDied","Data":"11dd445d26d9b3face72c5bc8d50ebb3db6ba80b3268d0dacdfe9fac9ac99709"} Nov 03 21:15:56.131086 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.131045 5932 generic.go:356] "Generic (PLEG): container finished" podID="660352ac-d467-4113-bbf4-f8b164da8f70" containerID="fa355741cd1f79cc395b04ccccf7baa74ce38fe4c8ea5b0dc3db1a9ee563d7c2" exitCode=0 Nov 03 21:15:56.131444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.131322 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" event={"ID":"660352ac-d467-4113-bbf4-f8b164da8f70","Type":"ContainerDied","Data":"fa355741cd1f79cc395b04ccccf7baa74ce38fe4c8ea5b0dc3db1a9ee563d7c2"} Nov 03 21:15:56.131444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.131376 5932 scope.go:122] "RemoveContainer" containerID="4f4d4bb9189513dcd920d6df77ad6fe6727741cc8196f1991fa5261bea76cc83" Nov 03 21:15:56.194688 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.194660 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:15:56.290670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.290638 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:56.317207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317182 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317227 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317297 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca\") pod \"260b4425-00a8-421f-8b53-b41a2382ab86\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " Nov 03 21:15:56.317340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317327 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx\" (UniqueName: \"kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx\") pod \"260b4425-00a8-421f-8b53-b41a2382ab86\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " Nov 03 21:15:56.317528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317391 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317425 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317455 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317510 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca\") pod \"260b4425-00a8-421f-8b53-b41a2382ab86\" (UID: \"260b4425-00a8-421f-8b53-b41a2382ab86\") " Nov 03 21:15:56.317797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317559 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.317797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317567 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317691 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.317797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317751 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.317797 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317768 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.318075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317804 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.318075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317839 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs\") pod \"660352ac-d467-4113-bbf4-f8b164da8f70\" (UID: \"660352ac-d467-4113-bbf4-f8b164da8f70\") " Nov 03 21:15:56.318075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.317909 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.318258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318085 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca" pod "260b4425-00a8-421f-8b53-b41a2382ab86" (UID: "260b4425-00a8-421f-8b53-b41a2382ab86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:56.318258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318105 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:15:56.318495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318468 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.318615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318498 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.318615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318514 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.318615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318529 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.318615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318544 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660352ac-d467-4113-bbf4-f8b164da8f70-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.318615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.318558 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.320106 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.320058 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca" pod "260b4425-00a8-421f-8b53-b41a2382ab86" (UID: "260b4425-00a8-421f-8b53-b41a2382ab86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:15:56.320195 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.320147 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx" pod "260b4425-00a8-421f-8b53-b41a2382ab86" (UID: "260b4425-00a8-421f-8b53-b41a2382ab86"). InnerVolumeSpecName "kube-api-access-qk6gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:56.320351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.320323 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:56.320469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.320397 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:15:56.320606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.320585 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:15:56.320748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.320730 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs" pod "660352ac-d467-4113-bbf4-f8b164da8f70" (UID: "660352ac-d467-4113-bbf4-f8b164da8f70"). InnerVolumeSpecName "kube-api-access-854cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:15:56.420575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.420537 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660352ac-d467-4113-bbf4-f8b164da8f70-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.420575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.420576 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660352ac-d467-4113-bbf4-f8b164da8f70-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.420821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.420592 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-854cs\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-kube-api-access-854cs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.420821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.420608 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660352ac-d467-4113-bbf4-f8b164da8f70-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.420821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.420637 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b4425-00a8-421f-8b53-b41a2382ab86-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:56.420821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:56.420650 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6gx\" (UniqueName: \"kubernetes.io/projected/260b4425-00a8-421f-8b53-b41a2382ab86-kube-api-access-qk6gx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:15:57.140058 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.140023 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" Nov 03 21:15:57.140549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.140057 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn" event={"ID":"660352ac-d467-4113-bbf4-f8b164da8f70","Type":"ContainerDied","Data":"df687f57878d3a620a3fe4b9c46b08c240b6704cae3b82f1b987d39a2081b4e5"} Nov 03 21:15:57.140549 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.140111 5932 scope.go:122] "RemoveContainer" containerID="fa355741cd1f79cc395b04ccccf7baa74ce38fe4c8ea5b0dc3db1a9ee563d7c2" Nov 03 21:15:57.141762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.141734 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" Nov 03 21:15:57.141889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.141759 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx" event={"ID":"260b4425-00a8-421f-8b53-b41a2382ab86","Type":"ContainerDied","Data":"4f0e2d4c713abb03834d4f3a1c0d67c54ad8f16b8f62932db172c623ed851320"} Nov 03 21:15:57.141889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.141837 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0e2d4c713abb03834d4f3a1c0d67c54ad8f16b8f62932db172c623ed851320" Nov 03 21:15:57.341291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.341220 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn"] Nov 03 21:15:57.346394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:57.346370 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-10fa59-openshift-default-6577f4d9f5-gdbmn"] Nov 03 21:15:58.051783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:15:58.051752 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" path="/var/lib/kubelet/pods/660352ac-d467-4113-bbf4-f8b164da8f70/volumes" Nov 03 21:16:00.500930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:00.500901 5932 scope.go:122] "RemoveContainer" containerID="a2ba49b66994f08e2ae493025c77fd35153a6e0f3a2c7d2c916aaf17b0aacbd1" Nov 03 21:16:00.513083 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:00.513062 5932 scope.go:122] "RemoveContainer" containerID="9ecd1edc12b82bf53ff74eb7a622efa94687cbfd9b7e8672f6039927392191b6" Nov 03 21:16:01.046861 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:01.046825 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn"] Nov 03 21:16:01.093001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:01.092956 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9044df0d-5n9qn"] Nov 03 21:16:02.052965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:02.052928 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8288226b-394d-450a-834e-e68b5f402aab" path="/var/lib/kubelet/pods/8288226b-394d-450a-834e-e68b5f402aab/volumes" Nov 03 21:16:05.766548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:05.766506 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh"] Nov 03 21:16:05.812495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:05.812461 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-cfcd54-hqbqh"] Nov 03 21:16:05.946463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:05.946425 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69"] Nov 03 21:16:06.056783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:06.056698 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432322dd-419d-4d92-ae14-d02696d404c4" path="/var/lib/kubelet/pods/432322dd-419d-4d92-ae14-d02696d404c4/volumes" Nov 03 21:16:06.066126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:06.066084 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj"] Nov 03 21:16:10.361994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.361946 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7"] Nov 03 21:16:10.362772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362749 5932 cpu_manager.go:414] "containerMap: removing container" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" Nov 03 21:16:10.362772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362769 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" Nov 03 21:16:10.362943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362802 5932 cpu_manager.go:414] "containerMap: removing container" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" Nov 03 21:16:10.362943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362808 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" Nov 03 21:16:10.362943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362817 5932 cpu_manager.go:414] "containerMap: removing container" podUID="260b4425-00a8-421f-8b53-b41a2382ab86" containerName="revoke-keys" Nov 03 21:16:10.362943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362822 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260b4425-00a8-421f-8b53-b41a2382ab86" containerName="revoke-keys" Nov 03 21:16:10.362943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362913 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" Nov 03 21:16:10.362943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.362925 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="260b4425-00a8-421f-8b53-b41a2382ab86" containerName="revoke-keys" Nov 03 21:16:10.368661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.368639 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.419579 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.419546 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7"] Nov 03 21:16:10.461405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.461377 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:10.461566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.461426 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:16:10.479594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.479562 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.479784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.479610 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.479784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.479692 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlvxf\" (UniqueName: \"kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.581527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.581478 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.581752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.581545 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.581752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.581565 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlvxf\" (UniqueName: \"kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.582348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.582328 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.584318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.584290 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.639406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.639376 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlvxf\" (UniqueName: \"kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf\") pod \"maas-api-revoke-sub-bee2d2ed-bvtc7\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.684425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.684392 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:10.947752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:10.947674 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" podUID="e946cf50-c4f9-48ec-b22f-3046232b4601" containerName="payload-processing" containerID="cri-o://91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3" gracePeriod=25 Nov 03 21:16:11.064944 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.064909 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7"] Nov 03 21:16:11.066445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.066393 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" podUID="fa65f1a0-75cc-4dfa-97b8-51c8a3089012" containerName="payload-pre-processing" containerID="cri-o://f28ef7dc96aa9d80ae8c908ec58d0ca965282d8f783402d1f2fa15cd59c5147a" gracePeriod=25 Nov 03 21:16:11.071743 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:16:11.071718 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19753c32_cfc3_4e4b_946c_ea7c50472ac4.slice/crio-c95ea017a63a6ca37e4ba16cf4424b4a409c2a3e3b99b2700df4abffcec0b0b6 WatchSource:0}: Error finding container c95ea017a63a6ca37e4ba16cf4424b4a409c2a3e3b99b2700df4abffcec0b0b6: Status 404 returned error can't find the container with id c95ea017a63a6ca37e4ba16cf4424b4a409c2a3e3b99b2700df4abffcec0b0b6 Nov 03 21:16:11.285522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.285186 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 21:16:11.286283 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.286249 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" event={"ID":"19753c32-cfc3-4e4b-946c-ea7c50472ac4","Type":"ContainerStarted","Data":"c95ea017a63a6ca37e4ba16cf4424b4a409c2a3e3b99b2700df4abffcec0b0b6"} Nov 03 21:16:11.289653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.289587 5932 generic.go:356] "Generic (PLEG): container finished" podID="e946cf50-c4f9-48ec-b22f-3046232b4601" containerID="91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3" exitCode=0 Nov 03 21:16:11.289811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.289791 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" event={"ID":"e946cf50-c4f9-48ec-b22f-3046232b4601","Type":"ContainerDied","Data":"91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3"} Nov 03 21:16:11.289890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.289842 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" event={"ID":"e946cf50-c4f9-48ec-b22f-3046232b4601","Type":"ContainerDied","Data":"54bc73089db535a572c76f42eb54ea5f3c88438a060bef6a3aab97c822895d4e"} Nov 03 21:16:11.289890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.289875 5932 scope.go:122] "RemoveContainer" containerID="91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3" Nov 03 21:16:11.290050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.290034 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69" Nov 03 21:16:11.296585 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.293683 5932 generic.go:356] "Generic (PLEG): container finished" podID="fa65f1a0-75cc-4dfa-97b8-51c8a3089012" containerID="f28ef7dc96aa9d80ae8c908ec58d0ca965282d8f783402d1f2fa15cd59c5147a" exitCode=0 Nov 03 21:16:11.296585 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.293735 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" event={"ID":"fa65f1a0-75cc-4dfa-97b8-51c8a3089012","Type":"ContainerDied","Data":"f28ef7dc96aa9d80ae8c908ec58d0ca965282d8f783402d1f2fa15cd59c5147a"} Nov 03 21:16:11.304717 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.304655 5932 scope.go:122] "RemoveContainer" containerID="91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3" Nov 03 21:16:11.305280 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:11.305097 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3\": container with ID starting with 91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3 not found: ID does not exist" containerID="91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3" Nov 03 21:16:11.305280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.305137 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3"} err="failed to get container status \"91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3\": rpc error: code = NotFound desc = could not find container \"91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3\": container with ID starting with 91e8db8c58207382d862279bf54e67a601036276b7999f128bcc26d41c3ed6f3 not found: ID does not exist" Nov 03 21:16:11.376572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.376548 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 21:16:11.391932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.391900 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume\") pod \"e946cf50-c4f9-48ec-b22f-3046232b4601\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " Nov 03 21:16:11.392110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.392093 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz\" (UniqueName: \"kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz\") pod \"e946cf50-c4f9-48ec-b22f-3046232b4601\" (UID: \"e946cf50-c4f9-48ec-b22f-3046232b4601\") " Nov 03 21:16:11.392321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.392296 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume" pod "e946cf50-c4f9-48ec-b22f-3046232b4601" (UID: "e946cf50-c4f9-48ec-b22f-3046232b4601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:11.392451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.392434 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e946cf50-c4f9-48ec-b22f-3046232b4601-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:11.394865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.394839 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz" pod "e946cf50-c4f9-48ec-b22f-3046232b4601" (UID: "e946cf50-c4f9-48ec-b22f-3046232b4601"). InnerVolumeSpecName "kube-api-access-fsfxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:11.493044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.492969 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb\" (UniqueName: \"kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb\") pod \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " Nov 03 21:16:11.493219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.493200 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume\") pod \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\" (UID: \"fa65f1a0-75cc-4dfa-97b8-51c8a3089012\") " Nov 03 21:16:11.493578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.493550 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume" pod "fa65f1a0-75cc-4dfa-97b8-51c8a3089012" (UID: "fa65f1a0-75cc-4dfa-97b8-51c8a3089012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:11.493713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.493695 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:11.493771 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.493716 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsfxz\" (UniqueName: \"kubernetes.io/projected/e946cf50-c4f9-48ec-b22f-3046232b4601-kube-api-access-fsfxz\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:11.495194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.495175 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb" pod "fa65f1a0-75cc-4dfa-97b8-51c8a3089012" (UID: "fa65f1a0-75cc-4dfa-97b8-51c8a3089012"). InnerVolumeSpecName "kube-api-access-wlcgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:11.594570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.594544 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlcgb\" (UniqueName: \"kubernetes.io/projected/fa65f1a0-75cc-4dfa-97b8-51c8a3089012-kube-api-access-wlcgb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:11.711341 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.711298 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69"] Nov 03 21:16:11.715003 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:11.714975 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3b8feb-5cf8bfbc87-jsf69"] Nov 03 21:16:12.056468 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.056428 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e946cf50-c4f9-48ec-b22f-3046232b4601" path="/var/lib/kubelet/pods/e946cf50-c4f9-48ec-b22f-3046232b4601/volumes" Nov 03 21:16:12.303475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.303443 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" Nov 03 21:16:12.303475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.303460 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj" event={"ID":"fa65f1a0-75cc-4dfa-97b8-51c8a3089012","Type":"ContainerDied","Data":"b37fd6f25577cb6703004d6dddee39447708b770df8955734dcd2a7da4fc74d8"} Nov 03 21:16:12.303812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.303557 5932 scope.go:122] "RemoveContainer" containerID="f28ef7dc96aa9d80ae8c908ec58d0ca965282d8f783402d1f2fa15cd59c5147a" Nov 03 21:16:12.306310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.306284 5932 generic.go:356] "Generic (PLEG): container finished" podID="19753c32-cfc3-4e4b-946c-ea7c50472ac4" containerID="8da7e1fb8282501fb59171688a7a333b83783799d60a7ad76c1149219270eac2" exitCode=0 Nov 03 21:16:12.306434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.306399 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" event={"ID":"19753c32-cfc3-4e4b-946c-ea7c50472ac4","Type":"ContainerDied","Data":"8da7e1fb8282501fb59171688a7a333b83783799d60a7ad76c1149219270eac2"} Nov 03 21:16:12.605013 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.604935 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj"] Nov 03 21:16:12.635694 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:12.635663 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-3b8feb-5dc44c9b6c-sqrkj"] Nov 03 21:16:13.493050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.493020 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:13.594298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.594207 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws"] Nov 03 21:16:13.594481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.594453 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" containerID="cri-o://98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf" gracePeriod=30 Nov 03 21:16:13.618933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.617887 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca\") pod \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " Nov 03 21:16:13.618933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.618041 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca\") pod \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " Nov 03 21:16:13.618933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.618077 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf\" (UniqueName: \"kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf\") pod \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\" (UID: \"19753c32-cfc3-4e4b-946c-ea7c50472ac4\") " Nov 03 21:16:13.618933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.618428 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca" pod "19753c32-cfc3-4e4b-946c-ea7c50472ac4" (UID: "19753c32-cfc3-4e4b-946c-ea7c50472ac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:13.618933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.618657 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:13.621367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.621340 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca" pod "19753c32-cfc3-4e4b-946c-ea7c50472ac4" (UID: "19753c32-cfc3-4e4b-946c-ea7c50472ac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:13.621544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.621522 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf" pod "19753c32-cfc3-4e4b-946c-ea7c50472ac4" (UID: "19753c32-cfc3-4e4b-946c-ea7c50472ac4"). InnerVolumeSpecName "kube-api-access-mlvxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:13.721645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.721271 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19753c32-cfc3-4e4b-946c-ea7c50472ac4-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:13.721645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:13.721313 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlvxf\" (UniqueName: \"kubernetes.io/projected/19753c32-cfc3-4e4b-946c-ea7c50472ac4-kube-api-access-mlvxf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:14.051880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:14.051850 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa65f1a0-75cc-4dfa-97b8-51c8a3089012" path="/var/lib/kubelet/pods/fa65f1a0-75cc-4dfa-97b8-51c8a3089012/volumes" Nov 03 21:16:14.152208 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:14.152172 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:16:14.326347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:14.326275 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" Nov 03 21:16:14.326347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:14.326303 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7" event={"ID":"19753c32-cfc3-4e4b-946c-ea7c50472ac4","Type":"ContainerDied","Data":"c95ea017a63a6ca37e4ba16cf4424b4a409c2a3e3b99b2700df4abffcec0b0b6"} Nov 03 21:16:14.326347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:14.326347 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95ea017a63a6ca37e4ba16cf4424b4a409c2a3e3b99b2700df4abffcec0b0b6" Nov 03 21:16:15.997074 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:15.997038 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6"] Nov 03 21:16:15.997523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:15.997253 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" podUID="1039978d-115e-4129-9f85-2a09f7afd1a6" containerName="maas-api" containerID="cri-o://0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26" gracePeriod=30 Nov 03 21:16:16.319018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.318989 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:16:16.343601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.343570 5932 generic.go:356] "Generic (PLEG): container finished" podID="1039978d-115e-4129-9f85-2a09f7afd1a6" containerID="0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26" exitCode=0 Nov 03 21:16:16.343819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.343613 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" Nov 03 21:16:16.343819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.343677 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" event={"ID":"1039978d-115e-4129-9f85-2a09f7afd1a6","Type":"ContainerDied","Data":"0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26"} Nov 03 21:16:16.343819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.343734 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6" event={"ID":"1039978d-115e-4129-9f85-2a09f7afd1a6","Type":"ContainerDied","Data":"dbab3238a74e39a432570cdb1541fc1cdda1fc431c1093076f49971336858bba"} Nov 03 21:16:16.343819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.343759 5932 scope.go:122] "RemoveContainer" containerID="0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26" Nov 03 21:16:16.354295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.354277 5932 scope.go:122] "RemoveContainer" containerID="0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26" Nov 03 21:16:16.354600 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:16.354569 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26\": container with ID starting with 0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26 not found: ID does not exist" containerID="0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26" Nov 03 21:16:16.354692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.354606 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26"} err="failed to get container status \"0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26\": rpc error: code = NotFound desc = could not find container \"0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26\": container with ID starting with 0194f0e1bfc54d0df9d0c5e4aa90e96294145d7fddf7dd67e4c003d4286d1a26 not found: ID does not exist" Nov 03 21:16:16.458379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.458336 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls\") pod \"1039978d-115e-4129-9f85-2a09f7afd1a6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " Nov 03 21:16:16.458578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.458458 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs\") pod \"1039978d-115e-4129-9f85-2a09f7afd1a6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " Nov 03 21:16:16.458578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.458509 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca\") pod \"1039978d-115e-4129-9f85-2a09f7afd1a6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " Nov 03 21:16:16.458752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.458687 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49\" (UniqueName: \"kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49\") pod \"1039978d-115e-4129-9f85-2a09f7afd1a6\" (UID: \"1039978d-115e-4129-9f85-2a09f7afd1a6\") " Nov 03 21:16:16.459197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.458966 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca" pod "1039978d-115e-4129-9f85-2a09f7afd1a6" (UID: "1039978d-115e-4129-9f85-2a09f7afd1a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:16.459197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.459162 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1039978d-115e-4129-9f85-2a09f7afd1a6-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:16.461161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.461121 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49" pod "1039978d-115e-4129-9f85-2a09f7afd1a6" (UID: "1039978d-115e-4129-9f85-2a09f7afd1a6"). InnerVolumeSpecName "kube-api-access-5fp49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:16.461701 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.461677 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs" pod "1039978d-115e-4129-9f85-2a09f7afd1a6" (UID: "1039978d-115e-4129-9f85-2a09f7afd1a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:16.461905 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.461887 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls" pod "1039978d-115e-4129-9f85-2a09f7afd1a6" (UID: "1039978d-115e-4129-9f85-2a09f7afd1a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:16.570613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.565590 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fp49\" (UniqueName: \"kubernetes.io/projected/1039978d-115e-4129-9f85-2a09f7afd1a6-kube-api-access-5fp49\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:16.570613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.565689 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:16.570613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.565709 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1039978d-115e-4129-9f85-2a09f7afd1a6-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:16.758335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.758296 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6"] Nov 03 21:16:16.762166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:16.762139 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df71d7-7d8d457659-cmqm6"] Nov 03 21:16:18.068230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.067526 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1039978d-115e-4129-9f85-2a09f7afd1a6" path="/var/lib/kubelet/pods/1039978d-115e-4129-9f85-2a09f7afd1a6/volumes" Nov 03 21:16:18.345543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.345462 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx"] Nov 03 21:16:18.391002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.390958 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-df71d7-9hpbx"] Nov 03 21:16:18.868617 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.868539 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:16:18.904948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.904907 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.904961 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.904998 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905057 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905113 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905244 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905248 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:18.905365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905296 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905333 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905472 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:18.905590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905549 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:18.905590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905555 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:18.905779 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905592 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket\") pod \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\" (UID: \"c4db7b2e-321b-4631-ad4f-10373b59ae8b\") " Nov 03 21:16:18.905857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.905833 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:18.906191 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.906171 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.906191 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.906190 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.906420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.906201 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.906420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.906209 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.906420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.906218 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:18.907309 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.907282 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:18.907648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.907603 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:18.907773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.907740 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:16:18.907773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:18.907761 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7" pod "c4db7b2e-321b-4631-ad4f-10373b59ae8b" (UID: "c4db7b2e-321b-4631-ad4f-10373b59ae8b"). InnerVolumeSpecName "kube-api-access-nwfl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:19.007291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.007255 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.007291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.007285 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.007291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.007296 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwfl7\" (UniqueName: \"kubernetes.io/projected/c4db7b2e-321b-4631-ad4f-10373b59ae8b-kube-api-access-nwfl7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.007540 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.007306 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4db7b2e-321b-4631-ad4f-10373b59ae8b-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:19.077296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.077258 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl"] Nov 03 21:16:19.369347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.369305 5932 generic.go:356] "Generic (PLEG): container finished" podID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerID="98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf" exitCode=0 Nov 03 21:16:19.369545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.369383 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" event={"ID":"c4db7b2e-321b-4631-ad4f-10373b59ae8b","Type":"ContainerDied","Data":"98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf"} Nov 03 21:16:19.369545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.369435 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" event={"ID":"c4db7b2e-321b-4631-ad4f-10373b59ae8b","Type":"ContainerDied","Data":"6eb8fa6ca900dd833c4cf37b80fb9acca324d04c1e3fcdc18ab25f44f2b3d469"} Nov 03 21:16:19.369545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.369456 5932 scope.go:122] "RemoveContainer" containerID="98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf" Nov 03 21:16:19.369545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.369397 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws" Nov 03 21:16:19.384089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.383726 5932 scope.go:122] "RemoveContainer" containerID="cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441" Nov 03 21:16:19.394134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.394100 5932 scope.go:122] "RemoveContainer" containerID="98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf" Nov 03 21:16:19.394379 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:19.394363 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf\": container with ID starting with 98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf not found: ID does not exist" containerID="98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf" Nov 03 21:16:19.394433 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.394388 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf"} err="failed to get container status \"98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf\": rpc error: code = NotFound desc = could not find container \"98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf\": container with ID starting with 98db4fb0f0740c282bc10b2a2e1e44e2f433f34286cdf97110271021382a28bf not found: ID does not exist" Nov 03 21:16:19.394433 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.394406 5932 scope.go:122] "RemoveContainer" containerID="cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441" Nov 03 21:16:19.394676 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:19.394660 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441\": container with ID starting with cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441 not found: ID does not exist" containerID="cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441" Nov 03 21:16:19.394740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.394680 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441"} err="failed to get container status \"cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441\": rpc error: code = NotFound desc = could not find container \"cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441\": container with ID starting with cb68ef095976b73f4a6b833ceed98d3e8942993f8d991f5ee50dca0fcf053441 not found: ID does not exist" Nov 03 21:16:19.524254 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.524222 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws"] Nov 03 21:16:19.529098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:19.529071 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cfcd54-openshift-default-7987d44477-cmfws"] Nov 03 21:16:20.052028 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:20.051995 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260b4425-00a8-421f-8b53-b41a2382ab86" path="/var/lib/kubelet/pods/260b4425-00a8-421f-8b53-b41a2382ab86/volumes" Nov 03 21:16:20.052485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:20.052471 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" path="/var/lib/kubelet/pods/c4db7b2e-321b-4631-ad4f-10373b59ae8b/volumes" Nov 03 21:16:24.970054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:24.970011 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn"] Nov 03 21:16:25.091805 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:25.091771 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl"] Nov 03 21:16:26.250538 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.250497 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb"] Nov 03 21:16:26.251121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251097 5932 cpu_manager.go:414] "containerMap: removing container" podUID="fa65f1a0-75cc-4dfa-97b8-51c8a3089012" containerName="payload-pre-processing" Nov 03 21:16:26.251121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251119 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa65f1a0-75cc-4dfa-97b8-51c8a3089012" containerName="payload-pre-processing" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251144 5932 cpu_manager.go:414] "containerMap: removing container" podUID="19753c32-cfc3-4e4b-946c-ea7c50472ac4" containerName="revoke-keys" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251150 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19753c32-cfc3-4e4b-946c-ea7c50472ac4" containerName="revoke-keys" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251165 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1039978d-115e-4129-9f85-2a09f7afd1a6" containerName="maas-api" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251170 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1039978d-115e-4129-9f85-2a09f7afd1a6" containerName="maas-api" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251196 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251201 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251211 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251216 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251229 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e946cf50-c4f9-48ec-b22f-3046232b4601" containerName="payload-processing" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251233 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e946cf50-c4f9-48ec-b22f-3046232b4601" containerName="payload-processing" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251311 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa65f1a0-75cc-4dfa-97b8-51c8a3089012" containerName="payload-pre-processing" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251324 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="19753c32-cfc3-4e4b-946c-ea7c50472ac4" containerName="revoke-keys" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251332 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1039978d-115e-4129-9f85-2a09f7afd1a6" containerName="maas-api" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251339 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251345 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4db7b2e-321b-4631-ad4f-10373b59ae8b" containerName="istio-proxy" Nov 03 21:16:26.251343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251351 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="660352ac-d467-4113-bbf4-f8b164da8f70" containerName="istio-proxy" Nov 03 21:16:26.251963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.251359 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e946cf50-c4f9-48ec-b22f-3046232b4601" containerName="payload-processing" Nov 03 21:16:26.266280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.266244 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.309126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.309090 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb"] Nov 03 21:16:26.315016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.314989 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:16:26.315016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.315002 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:26.392974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.392927 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.393175 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.392983 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.393175 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.393094 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.501671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.501560 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.501967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.501940 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.502177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.502156 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.503477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.503451 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.526244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.526077 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.561227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.561186 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97\") pod \"maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.580960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.580930 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:26.966295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:26.966256 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb"] Nov 03 21:16:26.972155 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:16:26.972124 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7686414_f8c9_4fc2_94ce_9d3a8341d7ed.slice/crio-29a2a0845a3c992d87715dd14e4c45321b963e86dec2888b112b5a7ab6bd91c6 WatchSource:0}: Error finding container 29a2a0845a3c992d87715dd14e4c45321b963e86dec2888b112b5a7ab6bd91c6: Status 404 returned error can't find the container with id 29a2a0845a3c992d87715dd14e4c45321b963e86dec2888b112b5a7ab6bd91c6 Nov 03 21:16:27.436391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:27.436349 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" event={"ID":"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed","Type":"ContainerStarted","Data":"fdab67556533f666e2840c21b33af44505c890b2475b4a9556db4a64342d547b"} Nov 03 21:16:27.436832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:27.436411 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" event={"ID":"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed","Type":"ContainerStarted","Data":"29a2a0845a3c992d87715dd14e4c45321b963e86dec2888b112b5a7ab6bd91c6"} Nov 03 21:16:27.547169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:27.547064 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" podStartSLOduration=1.547050316 podStartE2EDuration="1.547050316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:16:27.544429857 +0000 UTC m=+2795.609671336" watchObservedRunningTime="2026-11-03 21:16:27.547050316 +0000 UTC m=+2795.612291802" Nov 03 21:16:28.445258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:28.445218 5932 generic.go:356] "Generic (PLEG): container finished" podID="c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" containerID="fdab67556533f666e2840c21b33af44505c890b2475b4a9556db4a64342d547b" exitCode=0 Nov 03 21:16:28.445692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:28.445300 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" event={"ID":"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed","Type":"ContainerDied","Data":"fdab67556533f666e2840c21b33af44505c890b2475b4a9556db4a64342d547b"} Nov 03 21:16:29.604357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.604330 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:29.639718 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.639688 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97\") pod \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " Nov 03 21:16:29.639866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.639766 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca\") pod \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " Nov 03 21:16:29.639948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.639862 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca\") pod \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\" (UID: \"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed\") " Nov 03 21:16:29.640365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.640327 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca" pod "c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" (UID: "c7686414-f8c9-4fc2-94ce-9d3a8341d7ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:29.641867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.641840 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97" pod "c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" (UID: "c7686414-f8c9-4fc2-94ce-9d3a8341d7ed"). InnerVolumeSpecName "kube-api-access-rvz97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:29.641967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.641897 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca" pod "c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" (UID: "c7686414-f8c9-4fc2-94ce-9d3a8341d7ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:29.741275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.741191 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:29.741275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.741225 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvz97\" (UniqueName: \"kubernetes.io/projected/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-kube-api-access-rvz97\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:29.741275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.741236 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:29.971386 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:29.971342 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerName="payload-processing" containerID="cri-o://d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73" gracePeriod=25 Nov 03 21:16:30.092784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:30.092745 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerName="payload-pre-processing" containerID="cri-o://638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1" gracePeriod=25 Nov 03 21:16:30.460577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:30.460544 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" Nov 03 21:16:30.460789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:30.460585 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb" event={"ID":"c7686414-f8c9-4fc2-94ce-9d3a8341d7ed","Type":"ContainerDied","Data":"29a2a0845a3c992d87715dd14e4c45321b963e86dec2888b112b5a7ab6bd91c6"} Nov 03 21:16:30.460789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:30.460685 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a2a0845a3c992d87715dd14e4c45321b963e86dec2888b112b5a7ab6bd91c6" Nov 03 21:16:32.073576 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:32.073538 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:16:32.074071 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:32.073587 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:16:34.114738 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.107054 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" containerID="cri-o://131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c" gracePeriod=45 Nov 03 21:16:34.227763 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.227723 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" probeResult="failure" output="Get \"https://10.132.0.111:8000/ready\": dial tcp 10.132.0.111:8000: connect: connection refused" Nov 03 21:16:34.391851 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.391821 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 21:16:34.492575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.492535 5932 generic.go:356] "Generic (PLEG): container finished" podID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerID="131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c" exitCode=0 Nov 03 21:16:34.492826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.492593 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" Nov 03 21:16:34.492826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.492669 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" event={"ID":"d677dd53-9912-4d03-8a81-7174bf5f9b88","Type":"ContainerDied","Data":"131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c"} Nov 03 21:16:34.492826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.492731 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl" event={"ID":"d677dd53-9912-4d03-8a81-7174bf5f9b88","Type":"ContainerDied","Data":"d09d1b512970d7eb882026446bd0f4a34c7029322d7801c022d39f56e11e587c"} Nov 03 21:16:34.492826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.492761 5932 scope.go:122] "RemoveContainer" containerID="131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c" Nov 03 21:16:34.499475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.499441 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home\") pod \"d677dd53-9912-4d03-8a81-7174bf5f9b88\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " Nov 03 21:16:34.499747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.499716 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home" pod "d677dd53-9912-4d03-8a81-7174bf5f9b88" (UID: "d677dd53-9912-4d03-8a81-7174bf5f9b88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.499747 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.499735 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87\" (UniqueName: \"kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87\") pod \"d677dd53-9912-4d03-8a81-7174bf5f9b88\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " Nov 03 21:16:34.499945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.499800 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache\") pod \"d677dd53-9912-4d03-8a81-7174bf5f9b88\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " Nov 03 21:16:34.499945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.499857 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs\" (UniqueName: \"kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs\") pod \"d677dd53-9912-4d03-8a81-7174bf5f9b88\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " Nov 03 21:16:34.500067 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.499980 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm\") pod \"d677dd53-9912-4d03-8a81-7174bf5f9b88\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " Nov 03 21:16:34.500137 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.500080 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir\") pod \"d677dd53-9912-4d03-8a81-7174bf5f9b88\" (UID: \"d677dd53-9912-4d03-8a81-7174bf5f9b88\") " Nov 03 21:16:34.500660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.500638 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.500801 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.500774 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir" pod "d677dd53-9912-4d03-8a81-7174bf5f9b88" (UID: "d677dd53-9912-4d03-8a81-7174bf5f9b88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.500882 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.500850 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache" pod "d677dd53-9912-4d03-8a81-7174bf5f9b88" (UID: "d677dd53-9912-4d03-8a81-7174bf5f9b88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.502958 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.502897 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm" pod "d677dd53-9912-4d03-8a81-7174bf5f9b88" (UID: "d677dd53-9912-4d03-8a81-7174bf5f9b88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:16:34.503071 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.503034 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87" pod "d677dd53-9912-4d03-8a81-7174bf5f9b88" (UID: "d677dd53-9912-4d03-8a81-7174bf5f9b88"). InnerVolumeSpecName "kube-api-access-hqw87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:34.504196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.504169 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs" pod "d677dd53-9912-4d03-8a81-7174bf5f9b88" (UID: "d677dd53-9912-4d03-8a81-7174bf5f9b88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:34.507297 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.507274 5932 scope.go:122] "RemoveContainer" containerID="131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c" Nov 03 21:16:34.507743 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:34.507720 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c\": container with ID starting with 131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c not found: ID does not exist" containerID="131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c" Nov 03 21:16:34.507841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.507751 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c"} err="failed to get container status \"131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c\": rpc error: code = NotFound desc = could not find container \"131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c\": container with ID starting with 131b9773f1adebadc396d9db85eb1b9f850c31623c610b57831cdf5288576b2c not found: ID does not exist" Nov 03 21:16:34.601671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.601616 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.601671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.601671 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqw87\" (UniqueName: \"kubernetes.io/projected/d677dd53-9912-4d03-8a81-7174bf5f9b88-kube-api-access-hqw87\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.601893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.601682 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.601893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.601694 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d677dd53-9912-4d03-8a81-7174bf5f9b88-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.601893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.601705 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d677dd53-9912-4d03-8a81-7174bf5f9b88-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:34.913718 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.913677 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl"] Nov 03 21:16:34.919995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:34.919968 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-88228a/distinct-2-w3-88228a-kserve-56b86d5cd-vmzhl"] Nov 03 21:16:36.051523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:36.051487 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" path="/var/lib/kubelet/pods/d677dd53-9912-4d03-8a81-7174bf5f9b88/volumes" Nov 03 21:16:37.715199 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.715160 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b"] Nov 03 21:16:37.715921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.715903 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" containerName="revoke-keys" Nov 03 21:16:37.715978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.715922 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" containerName="revoke-keys" Nov 03 21:16:37.715978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.715949 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" Nov 03 21:16:37.715978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.715954 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" Nov 03 21:16:37.716081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.716044 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d677dd53-9912-4d03-8a81-7174bf5f9b88" containerName="main" Nov 03 21:16:37.716081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.716065 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" containerName="revoke-keys" Nov 03 21:16:37.720881 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.720863 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.772971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.772936 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b"] Nov 03 21:16:37.814741 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.814702 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:16:37.814926 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.814764 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:16:37.847782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.847745 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.847955 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.847913 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.848024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.847970 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2dr\" (UniqueName: \"kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.949401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.949368 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.949572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.949421 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2dr\" (UniqueName: \"kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.949572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.949520 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.950228 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.950203 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:37.951841 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:37.951815 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:38.008782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:38.008709 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2dr\" (UniqueName: \"kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr\") pod \"maas-api-revoke-sub-e96e7c3d-7cf5b\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:38.035272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:38.035240 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:38.464668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:38.464591 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b"] Nov 03 21:16:38.469959 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:16:38.469928 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d6458f_332d_44ef_bed7_c25058966053.slice/crio-d3e5419beffb833732c63fdaa55daae2e4cafd0c97e233ceb1a4512932ce1513 WatchSource:0}: Error finding container d3e5419beffb833732c63fdaa55daae2e4cafd0c97e233ceb1a4512932ce1513: Status 404 returned error can't find the container with id d3e5419beffb833732c63fdaa55daae2e4cafd0c97e233ceb1a4512932ce1513 Nov 03 21:16:38.527502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:38.527470 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" event={"ID":"b5d6458f-332d-44ef-bed7-c25058966053","Type":"ContainerStarted","Data":"d3e5419beffb833732c63fdaa55daae2e4cafd0c97e233ceb1a4512932ce1513"} Nov 03 21:16:39.537150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:39.536606 5932 generic.go:356] "Generic (PLEG): container finished" podID="b5d6458f-332d-44ef-bed7-c25058966053" containerID="937823ec70564ecfa573d8ee5ecb7f6e3106a15fd53886f141167704a288fbb9" exitCode=0 Nov 03 21:16:39.537150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:39.536927 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" event={"ID":"b5d6458f-332d-44ef-bed7-c25058966053","Type":"ContainerDied","Data":"937823ec70564ecfa573d8ee5ecb7f6e3106a15fd53886f141167704a288fbb9"} Nov 03 21:16:40.696775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.696749 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:40.801426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.801341 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr\" (UniqueName: \"kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr\") pod \"b5d6458f-332d-44ef-bed7-c25058966053\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " Nov 03 21:16:40.801426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.801392 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca\") pod \"b5d6458f-332d-44ef-bed7-c25058966053\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " Nov 03 21:16:40.801645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.801521 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca\") pod \"b5d6458f-332d-44ef-bed7-c25058966053\" (UID: \"b5d6458f-332d-44ef-bed7-c25058966053\") " Nov 03 21:16:40.801900 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.801873 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca" pod "b5d6458f-332d-44ef-bed7-c25058966053" (UID: "b5d6458f-332d-44ef-bed7-c25058966053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:40.803471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.803450 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr" pod "b5d6458f-332d-44ef-bed7-c25058966053" (UID: "b5d6458f-332d-44ef-bed7-c25058966053"). InnerVolumeSpecName "kube-api-access-nh2dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:40.803575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.803529 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca" pod "b5d6458f-332d-44ef-bed7-c25058966053" (UID: "b5d6458f-332d-44ef-bed7-c25058966053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:16:40.903569 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.903531 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.903569 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.903566 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh2dr\" (UniqueName: \"kubernetes.io/projected/b5d6458f-332d-44ef-bed7-c25058966053-kube-api-access-nh2dr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:40.903800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:40.903580 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5d6458f-332d-44ef-bed7-c25058966053-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:41.551893 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:41.551856 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" Nov 03 21:16:41.552085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:41.551893 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b" event={"ID":"b5d6458f-332d-44ef-bed7-c25058966053","Type":"ContainerDied","Data":"d3e5419beffb833732c63fdaa55daae2e4cafd0c97e233ceb1a4512932ce1513"} Nov 03 21:16:41.552085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:41.551966 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e5419beffb833732c63fdaa55daae2e4cafd0c97e233ceb1a4512932ce1513" Nov 03 21:16:42.073145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:42.073104 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:16:42.073572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:42.073195 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:16:44.151400 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:44.151364 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:16:45.121085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:45.121050 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r"] Nov 03 21:16:45.145172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:45.145140 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f89d43d-h858r"] Nov 03 21:16:46.051713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:46.051681 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c21127-680d-4632-b165-09d8abffa2f4" path="/var/lib/kubelet/pods/a4c21127-680d-4632-b165-09d8abffa2f4/volumes" Nov 03 21:16:46.052110 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:46.052005 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s"] Nov 03 21:16:46.091202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:46.091169 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d04ac2e-l2n8s"] Nov 03 21:16:48.051672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:48.051614 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b69a290-82dd-4133-b4e7-fe9debd670a1" path="/var/lib/kubelet/pods/0b69a290-82dd-4133-b4e7-fe9debd670a1/volumes" Nov 03 21:16:50.256787 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.256760 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 21:16:50.374049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.374020 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 21:16:50.436939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.436910 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb\" (UniqueName: \"kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb\") pod \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " Nov 03 21:16:50.437147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.436972 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume\") pod \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\" (UID: \"5193f5f2-13c8-4ea5-aaa3-618fa746beae\") " Nov 03 21:16:50.437343 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.437319 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume" pod "5193f5f2-13c8-4ea5-aaa3-618fa746beae" (UID: "5193f5f2-13c8-4ea5-aaa3-618fa746beae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:50.439067 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.439047 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb" pod "5193f5f2-13c8-4ea5-aaa3-618fa746beae" (UID: "5193f5f2-13c8-4ea5-aaa3-618fa746beae"). InnerVolumeSpecName "kube-api-access-2b5lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:50.537789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.537705 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7\" (UniqueName: \"kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7\") pod \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " Nov 03 21:16:50.537952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.537866 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume\") pod \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\" (UID: \"a026c0ac-78fd-4adf-be17-ef0b5a98619f\") " Nov 03 21:16:50.538201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.538176 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume" pod "a026c0ac-78fd-4adf-be17-ef0b5a98619f" (UID: "a026c0ac-78fd-4adf-be17-ef0b5a98619f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:16:50.538340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.538326 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a026c0ac-78fd-4adf-be17-ef0b5a98619f-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:50.538340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.538340 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5lb\" (UniqueName: \"kubernetes.io/projected/5193f5f2-13c8-4ea5-aaa3-618fa746beae-kube-api-access-2b5lb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:50.538468 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.538351 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5193f5f2-13c8-4ea5-aaa3-618fa746beae-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:50.539901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.539867 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7" pod "a026c0ac-78fd-4adf-be17-ef0b5a98619f" (UID: "a026c0ac-78fd-4adf-be17-ef0b5a98619f"). InnerVolumeSpecName "kube-api-access-ssbj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:16:50.614125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.614081 5932 generic.go:356] "Generic (PLEG): container finished" podID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerID="d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73" exitCode=0 Nov 03 21:16:50.614290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.614132 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" Nov 03 21:16:50.614290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.614166 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" event={"ID":"5193f5f2-13c8-4ea5-aaa3-618fa746beae","Type":"ContainerDied","Data":"d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73"} Nov 03 21:16:50.614290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.614226 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn" event={"ID":"5193f5f2-13c8-4ea5-aaa3-618fa746beae","Type":"ContainerDied","Data":"eb09e3518d02c44885b64ef0154b327a4a794bf05fff4cd6f4920cad143274bb"} Nov 03 21:16:50.614290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.614256 5932 scope.go:122] "RemoveContainer" containerID="d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73" Nov 03 21:16:50.616227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.616199 5932 generic.go:356] "Generic (PLEG): container finished" podID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerID="638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1" exitCode=0 Nov 03 21:16:50.616390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.616259 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" Nov 03 21:16:50.616390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.616282 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" event={"ID":"a026c0ac-78fd-4adf-be17-ef0b5a98619f","Type":"ContainerDied","Data":"638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1"} Nov 03 21:16:50.616390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.616335 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl" event={"ID":"a026c0ac-78fd-4adf-be17-ef0b5a98619f","Type":"ContainerDied","Data":"64e4486dd5ba5bd74c78bd6d04492070485e8a438aba83052f948c0fd4e14acc"} Nov 03 21:16:50.626476 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.626451 5932 scope.go:122] "RemoveContainer" containerID="d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73" Nov 03 21:16:50.626943 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:50.626776 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73\": container with ID starting with d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73 not found: ID does not exist" containerID="d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73" Nov 03 21:16:50.626943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.626804 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73"} err="failed to get container status \"d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73\": rpc error: code = NotFound desc = could not find container \"d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73\": container with ID starting with d0dcfba0df817d9f4c2c583a6a1b671a20a8f5179d59c748eeb1e5bc01a76a73 not found: ID does not exist" Nov 03 21:16:50.626943 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.626825 5932 scope.go:122] "RemoveContainer" containerID="638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1" Nov 03 21:16:50.636493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.636477 5932 scope.go:122] "RemoveContainer" containerID="638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1" Nov 03 21:16:50.636812 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:16:50.636790 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1\": container with ID starting with 638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1 not found: ID does not exist" containerID="638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1" Nov 03 21:16:50.636889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.636821 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1"} err="failed to get container status \"638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1\": rpc error: code = NotFound desc = could not find container \"638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1\": container with ID starting with 638607408c2e9bbeede54f9dd3083a91a3362a7830b93766258f052bb48946b1 not found: ID does not exist" Nov 03 21:16:50.639456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.639438 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssbj7\" (UniqueName: \"kubernetes.io/projected/a026c0ac-78fd-4adf-be17-ef0b5a98619f-kube-api-access-ssbj7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:16:50.768112 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.768077 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl"] Nov 03 21:16:50.771908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.771879 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-88228a-5f865749ff-h9pcl"] Nov 03 21:16:50.873907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.873827 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn"] Nov 03 21:16:50.877902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:50.877872 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-88228a-6b96c66cf-dj9dn"] Nov 03 21:16:52.052523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:52.052488 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" path="/var/lib/kubelet/pods/5193f5f2-13c8-4ea5-aaa3-618fa746beae/volumes" Nov 03 21:16:52.053043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:16:52.052937 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" path="/var/lib/kubelet/pods/a026c0ac-78fd-4adf-be17-ef0b5a98619f/volumes" Nov 03 21:17:00.139997 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.139960 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5"] Nov 03 21:17:00.140550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.140198 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" podUID="777f4471-9570-46a2-a547-35ab9f189bc5" containerName="maas-api" containerID="cri-o://5711ba510e7d0c8c4f9c9d4d5695e006995bb59f81d8dd1df17277072bce3822" gracePeriod=30 Nov 03 21:17:00.454077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.454048 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:17:00.469178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.469151 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca\") pod \"777f4471-9570-46a2-a547-35ab9f189bc5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " Nov 03 21:17:00.469310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.469184 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs\" (UniqueName: \"kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs\") pod \"777f4471-9570-46a2-a547-35ab9f189bc5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " Nov 03 21:17:00.469310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.469240 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs\") pod \"777f4471-9570-46a2-a547-35ab9f189bc5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " Nov 03 21:17:00.469310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.469284 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls\") pod \"777f4471-9570-46a2-a547-35ab9f189bc5\" (UID: \"777f4471-9570-46a2-a547-35ab9f189bc5\") " Nov 03 21:17:00.469500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.469478 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca" pod "777f4471-9570-46a2-a547-35ab9f189bc5" (UID: "777f4471-9570-46a2-a547-35ab9f189bc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:00.469762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.469742 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/777f4471-9570-46a2-a547-35ab9f189bc5-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:00.471464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.471431 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs" pod "777f4471-9570-46a2-a547-35ab9f189bc5" (UID: "777f4471-9570-46a2-a547-35ab9f189bc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:00.471464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.471446 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls" pod "777f4471-9570-46a2-a547-35ab9f189bc5" (UID: "777f4471-9570-46a2-a547-35ab9f189bc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:00.471680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.471507 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs" pod "777f4471-9570-46a2-a547-35ab9f189bc5" (UID: "777f4471-9570-46a2-a547-35ab9f189bc5"). InnerVolumeSpecName "kube-api-access-prhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:00.570869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.570836 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prhhs\" (UniqueName: \"kubernetes.io/projected/777f4471-9570-46a2-a547-35ab9f189bc5-kube-api-access-prhhs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:00.570869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.570870 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:00.571073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.570882 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/777f4471-9570-46a2-a547-35ab9f189bc5-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:00.582560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.582530 5932 scope.go:122] "RemoveContainer" containerID="81e5a78d5f35ae74b4f4ceb12be6a765905d68a7deee362b168ee18b4393046f" Nov 03 21:17:00.595604 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.595577 5932 scope.go:122] "RemoveContainer" containerID="5711ba510e7d0c8c4f9c9d4d5695e006995bb59f81d8dd1df17277072bce3822" Nov 03 21:17:00.698355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.695974 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" Nov 03 21:17:00.713848 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.713796 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" event={"ID":"777f4471-9570-46a2-a547-35ab9f189bc5","Type":"ContainerDied","Data":"5711ba510e7d0c8c4f9c9d4d5695e006995bb59f81d8dd1df17277072bce3822"} Nov 03 21:17:00.714067 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.713877 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5" event={"ID":"777f4471-9570-46a2-a547-35ab9f189bc5","Type":"ContainerDied","Data":"165f498cde804ad1388dc9d74e19d617f143b9e8c0a8f9ef1151bf290f1cb628"} Nov 03 21:17:00.928898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.928865 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5"] Nov 03 21:17:00.935128 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:00.935101 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fc4e8d-6656f54f67-gkcs5"] Nov 03 21:17:02.054594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:02.054556 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777f4471-9570-46a2-a547-35ab9f189bc5" path="/var/lib/kubelet/pods/777f4471-9570-46a2-a547-35ab9f189bc5/volumes" Nov 03 21:17:02.348018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:02.347933 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb"] Nov 03 21:17:02.394170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:02.394136 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fc4e8d-sqwjb"] Nov 03 21:17:04.051889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:04.051856 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7686414-f8c9-4fc2-94ce-9d3a8341d7ed" path="/var/lib/kubelet/pods/c7686414-f8c9-4fc2-94ce-9d3a8341d7ed/volumes" Nov 03 21:17:09.733550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:09.733516 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w"] Nov 03 21:17:09.733994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:09.733761 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" containerID="cri-o://810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02" gracePeriod=30 Nov 03 21:17:10.731130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731088 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r"] Nov 03 21:17:10.731648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731604 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerName="payload-pre-processing" Nov 03 21:17:10.731648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731633 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerName="payload-pre-processing" Nov 03 21:17:10.731770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731654 5932 cpu_manager.go:414] "containerMap: removing container" podUID="777f4471-9570-46a2-a547-35ab9f189bc5" containerName="maas-api" Nov 03 21:17:10.731770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731660 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777f4471-9570-46a2-a547-35ab9f189bc5" containerName="maas-api" Nov 03 21:17:10.731770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731671 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b5d6458f-332d-44ef-bed7-c25058966053" containerName="revoke-keys" Nov 03 21:17:10.731770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731676 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d6458f-332d-44ef-bed7-c25058966053" containerName="revoke-keys" Nov 03 21:17:10.731770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731705 5932 cpu_manager.go:414] "containerMap: removing container" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerName="payload-processing" Nov 03 21:17:10.731770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731710 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerName="payload-processing" Nov 03 21:17:10.732012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731798 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="5193f5f2-13c8-4ea5-aaa3-618fa746beae" containerName="payload-processing" Nov 03 21:17:10.732012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731808 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a026c0ac-78fd-4adf-be17-ef0b5a98619f" containerName="payload-pre-processing" Nov 03 21:17:10.732012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731817 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="777f4471-9570-46a2-a547-35ab9f189bc5" containerName="maas-api" Nov 03 21:17:10.732012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.731822 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d6458f-332d-44ef-bed7-c25058966053" containerName="revoke-keys" Nov 03 21:17:10.737180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.737157 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.789651 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.789599 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.789856 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.789835 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwd5\" (UniqueName: \"kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.789967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.789946 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.790011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.789993 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r"] Nov 03 21:17:10.832116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.832076 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:10.832292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.832126 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:17:10.890891 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.890845 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.891073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.890955 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwd5\" (UniqueName: \"kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.891073 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.891003 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.891742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.891715 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.893263 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.893241 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:10.948169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:10.948133 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwd5\" (UniqueName: \"kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5\") pod \"maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:11.051057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.050976 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:11.428751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.428714 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r"] Nov 03 21:17:11.434912 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:17:11.434880 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbfd091_d5cd_4e17_92f2_8fbcf3844e2f.slice/crio-afe972d944848da35dab03f6f600ba3d998d968651f7b6948533ad584356b180 WatchSource:0}: Error finding container afe972d944848da35dab03f6f600ba3d998d968651f7b6948533ad584356b180: Status 404 returned error can't find the container with id afe972d944848da35dab03f6f600ba3d998d968651f7b6948533ad584356b180 Nov 03 21:17:11.679810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.679676 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb"] Nov 03 21:17:11.728001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.727972 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569"] Nov 03 21:17:11.728210 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.728183 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" containerID="cri-o://40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017" gracePeriod=30 Nov 03 21:17:11.733614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.733588 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6"] Nov 03 21:17:11.733867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.733831 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" containerID="cri-o://26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3" gracePeriod=30 Nov 03 21:17:11.736670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.736594 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb"] Nov 03 21:17:11.736880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.736855 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" containerID="cri-o://493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887" gracePeriod=30 Nov 03 21:17:11.772810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.772773 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" event={"ID":"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f","Type":"ContainerStarted","Data":"afe972d944848da35dab03f6f600ba3d998d968651f7b6948533ad584356b180"} Nov 03 21:17:11.773175 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:11.772990 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" containerID="cri-o://7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08" gracePeriod=30 Nov 03 21:17:12.781312 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:12.781228 5932 generic.go:356] "Generic (PLEG): container finished" podID="0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" containerID="4c79d907b4ba7bc63811c3a9b5623afed8aebe55bc8ec297356d950145b2f92b" exitCode=0 Nov 03 21:17:12.781700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:12.781307 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" event={"ID":"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f","Type":"ContainerDied","Data":"4c79d907b4ba7bc63811c3a9b5623afed8aebe55bc8ec297356d950145b2f92b"} Nov 03 21:17:13.936810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:13.936783 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:14.031012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.030972 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca\") pod \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " Nov 03 21:17:14.031219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.031018 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca\") pod \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " Nov 03 21:17:14.031219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.031105 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5\" (UniqueName: \"kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5\") pod \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\" (UID: \"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f\") " Nov 03 21:17:14.031422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.031394 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca" pod "0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" (UID: "0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:14.031528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.031513 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:14.033279 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.033250 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca" pod "0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" (UID: "0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:14.033393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.033312 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5" pod "0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" (UID: "0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f"). InnerVolumeSpecName "kube-api-access-mrwd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:14.132634 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.132598 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:14.132634 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.132640 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrwd5\" (UniqueName: \"kubernetes.io/projected/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f-kube-api-access-mrwd5\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:14.152529 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:14.152480 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:17:14.795271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.795245 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" Nov 03 21:17:14.795271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.795255 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r" event={"ID":"0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f","Type":"ContainerDied","Data":"afe972d944848da35dab03f6f600ba3d998d968651f7b6948533ad584356b180"} Nov 03 21:17:14.795485 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:14.795307 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afe972d944848da35dab03f6f600ba3d998d968651f7b6948533ad584356b180" Nov 03 21:17:15.015364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.015309 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:17:15.042760 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042735 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.042917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042805 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.042917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042826 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.042917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042869 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.042917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042901 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.043131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042935 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.043131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.042995 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.043131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043036 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.043131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043095 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs\") pod \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\" (UID: \"d7ab015e-f0c9-4936-8734-d1964c9cd4f1\") " Nov 03 21:17:15.043363 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043179 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:15.043363 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043187 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:15.043363 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043290 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:15.043533 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043420 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:15.043602 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043552 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:15.043766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043747 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.043833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043767 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.043833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043777 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.043833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043785 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.043833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.043794 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.045161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.045138 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:15.045275 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.045257 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:15.045507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.045479 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "kube-api-access-wkfpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:15.045615 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.045539 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo" pod "d7ab015e-f0c9-4936-8734-d1964c9cd4f1" (UID: "d7ab015e-f0c9-4936-8734-d1964c9cd4f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:15.145250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.145215 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.145250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.145244 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfpb\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-kube-api-access-wkfpb\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.145250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.145255 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.145608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.145265 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab015e-f0c9-4936-8734-d1964c9cd4f1-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:15.159196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.159143 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf"] Nov 03 21:17:15.166386 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.166356 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2"] Nov 03 21:17:15.185684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.185646 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf"] Nov 03 21:17:15.803837 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.803802 5932 generic.go:356] "Generic (PLEG): container finished" podID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerID="810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02" exitCode=0 Nov 03 21:17:15.804049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.803851 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" Nov 03 21:17:15.804049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.803900 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" event={"ID":"d7ab015e-f0c9-4936-8734-d1964c9cd4f1","Type":"ContainerDied","Data":"810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02"} Nov 03 21:17:15.804049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.803954 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w" event={"ID":"d7ab015e-f0c9-4936-8734-d1964c9cd4f1","Type":"ContainerDied","Data":"69c02292bd11865f08f97f81198f5ef7f47c3ae46497239f3add4f1cd8f6ef1d"} Nov 03 21:17:15.804049 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.803980 5932 scope.go:122] "RemoveContainer" containerID="810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02" Nov 03 21:17:15.815170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.815150 5932 scope.go:122] "RemoveContainer" containerID="5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401" Nov 03 21:17:15.824723 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.824705 5932 scope.go:122] "RemoveContainer" containerID="810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02" Nov 03 21:17:15.825012 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:15.824991 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02\": container with ID starting with 810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02 not found: ID does not exist" containerID="810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02" Nov 03 21:17:15.825069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.825022 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02"} err="failed to get container status \"810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02\": rpc error: code = NotFound desc = could not find container \"810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02\": container with ID starting with 810b4ac98d084ce2073ed2521cd857bdeb616b3193d1cc302a5c4089092dba02 not found: ID does not exist" Nov 03 21:17:15.825069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.825043 5932 scope.go:122] "RemoveContainer" containerID="5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401" Nov 03 21:17:15.825292 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:15.825276 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401\": container with ID starting with 5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401 not found: ID does not exist" containerID="5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401" Nov 03 21:17:15.825336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.825297 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401"} err="failed to get container status \"5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401\": rpc error: code = NotFound desc = could not find container \"5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401\": container with ID starting with 5a30ed278c7039e9cb50b49f48533e8ce7d7e5fd735f67752253d178ce04c401 not found: ID does not exist" Nov 03 21:17:15.963368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.963336 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w"] Nov 03 21:17:15.967389 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:15.967362 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fc4e8d-openshift-default-78f6c497d5-9wz7w"] Nov 03 21:17:16.051152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:16.051123 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" path="/var/lib/kubelet/pods/d7ab015e-f0c9-4936-8734-d1964c9cd4f1/volumes" Nov 03 21:17:17.047808 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.047685 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:17:17.081895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.081856 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:17:17.106871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.106843 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:17:17.119450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.119419 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:17:17.167097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167068 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.167294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167105 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.167294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167133 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.167294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167167 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.167294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167184 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.167294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167213 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167370 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167453 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167473 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167478 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167491 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167495 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.167589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167523 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167611 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167670 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167769 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167802 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167829 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167880 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167885 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167927 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167952 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.167985 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168037 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168028 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168078 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168115 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168153 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168202 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168235 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168265 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168317 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168357 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168385 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168416 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168443 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168473 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168532 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token\") pod \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\" (UID: \"8e55545e-c3ca-4550-86f8-ba0b7b12c513\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168587 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.168663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168641 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data\") pod \"393d8b8c-f4e6-4e66-b329-8cad52013719\" (UID: \"393d8b8c-f4e6-4e66-b329-8cad52013719\") " Nov 03 21:17:17.169658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168672 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.169658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168795 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.169658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168839 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy\") pod \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\" (UID: \"4c454362-f7e1-4ef6-94dd-fb29158f12b9\") " Nov 03 21:17:17.169658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.168885 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s\") pod \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\" (UID: \"1f1798e6-5adf-4c58-9409-869c2ffd2d38\") " Nov 03 21:17:17.169658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169430 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169748 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169770 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169784 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169799 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169813 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169828 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.169932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.169841 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.170285 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170105 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.170285 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170132 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.170722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170433 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:17.170722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170501 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.170722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170635 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.170722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170658 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:17.171055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170842 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.171055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170862 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.171055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.170902 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.171055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.171019 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.171817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.171362 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.171817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.171420 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.171817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.171565 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.171817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.171660 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:17.172330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.172302 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "kube-api-access-hzg7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.172978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.172951 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:17.173157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.173131 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.173524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.173478 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:17.173940 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.173904 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy" pod "393d8b8c-f4e6-4e66-b329-8cad52013719" (UID: "393d8b8c-f4e6-4e66-b329-8cad52013719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.174201 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.174172 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "kube-api-access-cxm7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.174295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.174232 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.174339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.174316 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.175038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.175001 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:17.175129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.175097 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:17.175129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.175099 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token" pod "8e55545e-c3ca-4550-86f8-ba0b7b12c513" (UID: "8e55545e-c3ca-4550-86f8-ba0b7b12c513"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.175357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.175339 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s" pod "1f1798e6-5adf-4c58-9409-869c2ffd2d38" (UID: "1f1798e6-5adf-4c58-9409-869c2ffd2d38"). InnerVolumeSpecName "kube-api-access-cpb4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.175459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.175444 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "kube-api-access-6lp58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:17.175941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.175923 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo" pod "4c454362-f7e1-4ef6-94dd-fb29158f12b9" (UID: "4c454362-f7e1-4ef6-94dd-fb29158f12b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271261 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271288 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271298 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lp58\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-kube-api-access-6lp58\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271308 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271318 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271327 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271336 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271346 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271355 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271363 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzg7t\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-kube-api-access-hzg7t\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271372 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271380 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271389 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxm7m\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-kube-api-access-cxm7m\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271398 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271405 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271413 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271422 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393d8b8c-f4e6-4e66-b329-8cad52013719-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271430 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271438 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271447 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271457 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c454362-f7e1-4ef6-94dd-fb29158f12b9-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271465 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpb4s\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-kube-api-access-cpb4s\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271473 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271481 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271490 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271500 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271508 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1798e6-5adf-4c58-9409-869c2ffd2d38-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271516 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e55545e-c3ca-4550-86f8-ba0b7b12c513-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.271695 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.271524 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393d8b8c-f4e6-4e66-b329-8cad52013719-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:17.819889 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.819852 5932 generic.go:356] "Generic (PLEG): container finished" podID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerID="40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017" exitCode=0 Nov 03 21:17:17.820043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.819909 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" Nov 03 21:17:17.820043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.819934 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" event={"ID":"1f1798e6-5adf-4c58-9409-869c2ffd2d38","Type":"ContainerDied","Data":"40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017"} Nov 03 21:17:17.820043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.820006 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569" event={"ID":"1f1798e6-5adf-4c58-9409-869c2ffd2d38","Type":"ContainerDied","Data":"b9b992e2c6203164c926ba589387cafe271ff50b243a9b91ab32f6e74cd9d6f9"} Nov 03 21:17:17.820043 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.820030 5932 scope.go:122] "RemoveContainer" containerID="40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017" Nov 03 21:17:17.821583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.821554 5932 generic.go:356] "Generic (PLEG): container finished" podID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerID="26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3" exitCode=0 Nov 03 21:17:17.821734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.821607 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" Nov 03 21:17:17.821734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.821664 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" event={"ID":"8e55545e-c3ca-4550-86f8-ba0b7b12c513","Type":"ContainerDied","Data":"26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3"} Nov 03 21:17:17.821734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.821710 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6" event={"ID":"8e55545e-c3ca-4550-86f8-ba0b7b12c513","Type":"ContainerDied","Data":"8516c3c77194e067673a61db7241e2ba00dfb8370075a661a0d3f775ed998f60"} Nov 03 21:17:17.823350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.823325 5932 generic.go:356] "Generic (PLEG): container finished" podID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerID="493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887" exitCode=0 Nov 03 21:17:17.823484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.823378 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" Nov 03 21:17:17.823484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.823411 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" event={"ID":"393d8b8c-f4e6-4e66-b329-8cad52013719","Type":"ContainerDied","Data":"493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887"} Nov 03 21:17:17.823484 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.823457 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb" event={"ID":"393d8b8c-f4e6-4e66-b329-8cad52013719","Type":"ContainerDied","Data":"fca1744288c9f27d9617984405542c1cf4631443bf5f3a8fa904b317386d3808"} Nov 03 21:17:17.825095 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.825072 5932 generic.go:356] "Generic (PLEG): container finished" podID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerID="7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08" exitCode=0 Nov 03 21:17:17.825225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.825116 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" Nov 03 21:17:17.825225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.825156 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" event={"ID":"4c454362-f7e1-4ef6-94dd-fb29158f12b9","Type":"ContainerDied","Data":"7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08"} Nov 03 21:17:17.825225 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.825204 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb" event={"ID":"4c454362-f7e1-4ef6-94dd-fb29158f12b9","Type":"ContainerDied","Data":"26d51fc019d709e4bc1a5d2ddee24a448797a84958eac1f92dab52accfa74bc4"} Nov 03 21:17:17.832763 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.832744 5932 scope.go:122] "RemoveContainer" containerID="40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017" Nov 03 21:17:17.833115 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:17.833092 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017\": container with ID starting with 40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017 not found: ID does not exist" containerID="40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017" Nov 03 21:17:17.833183 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.833126 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017"} err="failed to get container status \"40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017\": rpc error: code = NotFound desc = could not find container \"40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017\": container with ID starting with 40e7bdb9793167baf2e393f47d41225ac697cab0b57788169daaeec428312017 not found: ID does not exist" Nov 03 21:17:17.833183 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.833150 5932 scope.go:122] "RemoveContainer" containerID="26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3" Nov 03 21:17:17.844977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.844957 5932 scope.go:122] "RemoveContainer" containerID="26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3" Nov 03 21:17:17.845235 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:17.845218 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3\": container with ID starting with 26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3 not found: ID does not exist" containerID="26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3" Nov 03 21:17:17.845328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.845241 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3"} err="failed to get container status \"26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3\": rpc error: code = NotFound desc = could not find container \"26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3\": container with ID starting with 26fc1275250afc20a380cb5cc4cc7d23a5450f46894f75ed003f60a0039a74a3 not found: ID does not exist" Nov 03 21:17:17.845328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.845257 5932 scope.go:122] "RemoveContainer" containerID="493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887" Nov 03 21:17:17.857482 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.857462 5932 scope.go:122] "RemoveContainer" containerID="493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887" Nov 03 21:17:17.857866 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:17.857832 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887\": container with ID starting with 493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887 not found: ID does not exist" containerID="493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887" Nov 03 21:17:17.857949 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.857870 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887"} err="failed to get container status \"493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887\": rpc error: code = NotFound desc = could not find container \"493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887\": container with ID starting with 493d86ae40929e57d8670579d4cbf53c36b1d95119b84c3f705c25a38fb34887 not found: ID does not exist" Nov 03 21:17:17.857949 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.857889 5932 scope.go:122] "RemoveContainer" containerID="7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08" Nov 03 21:17:17.867227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.867211 5932 scope.go:122] "RemoveContainer" containerID="7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08" Nov 03 21:17:17.867497 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:17.867474 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08\": container with ID starting with 7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08 not found: ID does not exist" containerID="7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08" Nov 03 21:17:17.867588 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.867502 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08"} err="failed to get container status \"7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08\": rpc error: code = NotFound desc = could not find container \"7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08\": container with ID starting with 7f2018500fc0a29b1902fca4256a1946f7a7de0fcfcc0b7e3034c0ba6ef13c08 not found: ID does not exist" Nov 03 21:17:17.974965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:17.974928 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb"] Nov 03 21:17:18.021863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.021767 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9tvgb"] Nov 03 21:17:18.052033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.051999 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" path="/var/lib/kubelet/pods/393d8b8c-f4e6-4e66-b329-8cad52013719/volumes" Nov 03 21:17:18.124363 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.124323 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6"] Nov 03 21:17:18.128688 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.128660 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-4xld6"] Nov 03 21:17:18.231425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.231392 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569"] Nov 03 21:17:18.238121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.238090 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rz569"] Nov 03 21:17:18.341184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.341101 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb"] Nov 03 21:17:18.343564 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:18.343538 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nvmvb"] Nov 03 21:17:20.053080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:20.053044 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" path="/var/lib/kubelet/pods/1f1798e6-5adf-4c58-9409-869c2ffd2d38/volumes" Nov 03 21:17:20.053500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:20.053486 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" path="/var/lib/kubelet/pods/4c454362-f7e1-4ef6-94dd-fb29158f12b9/volumes" Nov 03 21:17:20.053934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:20.053920 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" path="/var/lib/kubelet/pods/8e55545e-c3ca-4550-86f8-ba0b7b12c513/volumes" Nov 03 21:17:22.257636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:22.257581 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k"] Nov 03 21:17:22.372113 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:22.372061 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp"] Nov 03 21:17:27.258230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.258156 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" podUID="e8d201df-b678-414a-92c6-38192347216d" containerName="payload-processing" containerID="cri-o://c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd" gracePeriod=25 Nov 03 21:17:27.373130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.373098 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" podUID="d2d6912c-89d8-414e-ba8c-d3bec81b13b1" containerName="payload-pre-processing" containerID="cri-o://21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c" gracePeriod=25 Nov 03 21:17:27.523509 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.523443 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 21:17:27.645365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.645337 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 21:17:27.685533 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.685505 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl\" (UniqueName: \"kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl\") pod \"e8d201df-b678-414a-92c6-38192347216d\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " Nov 03 21:17:27.685702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.685552 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume\") pod \"e8d201df-b678-414a-92c6-38192347216d\" (UID: \"e8d201df-b678-414a-92c6-38192347216d\") " Nov 03 21:17:27.685972 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.685945 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume" pod "e8d201df-b678-414a-92c6-38192347216d" (UID: "e8d201df-b678-414a-92c6-38192347216d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:27.687576 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.687557 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl" pod "e8d201df-b678-414a-92c6-38192347216d" (UID: "e8d201df-b678-414a-92c6-38192347216d"). InnerVolumeSpecName "kube-api-access-98gtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:27.786932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.786853 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk\" (UniqueName: \"kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk\") pod \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " Nov 03 21:17:27.787078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.786961 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume\") pod \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\" (UID: \"d2d6912c-89d8-414e-ba8c-d3bec81b13b1\") " Nov 03 21:17:27.787350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.787326 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume" pod "d2d6912c-89d8-414e-ba8c-d3bec81b13b1" (UID: "d2d6912c-89d8-414e-ba8c-d3bec81b13b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:27.787396 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.787338 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98gtl\" (UniqueName: \"kubernetes.io/projected/e8d201df-b678-414a-92c6-38192347216d-kube-api-access-98gtl\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:27.787396 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.787377 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8d201df-b678-414a-92c6-38192347216d-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:27.789094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.789077 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk" pod "d2d6912c-89d8-414e-ba8c-d3bec81b13b1" (UID: "d2d6912c-89d8-414e-ba8c-d3bec81b13b1"). InnerVolumeSpecName "kube-api-access-vq7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:27.888216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.888184 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7jk\" (UniqueName: \"kubernetes.io/projected/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-kube-api-access-vq7jk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:27.888216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.888211 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2d6912c-89d8-414e-ba8c-d3bec81b13b1-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:27.899059 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.899027 5932 generic.go:356] "Generic (PLEG): container finished" podID="d2d6912c-89d8-414e-ba8c-d3bec81b13b1" containerID="21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c" exitCode=0 Nov 03 21:17:27.899203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.899081 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" Nov 03 21:17:27.899203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.899122 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" event={"ID":"d2d6912c-89d8-414e-ba8c-d3bec81b13b1","Type":"ContainerDied","Data":"21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c"} Nov 03 21:17:27.899203 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.899185 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp" event={"ID":"d2d6912c-89d8-414e-ba8c-d3bec81b13b1","Type":"ContainerDied","Data":"687b76d47570cc27dbd0d46e676b92ca1911decc4aaf934bd193957a6ba89295"} Nov 03 21:17:27.899359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.899220 5932 scope.go:122] "RemoveContainer" containerID="21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c" Nov 03 21:17:27.900671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.900646 5932 generic.go:356] "Generic (PLEG): container finished" podID="e8d201df-b678-414a-92c6-38192347216d" containerID="c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd" exitCode=0 Nov 03 21:17:27.900785 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.900702 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" Nov 03 21:17:27.900785 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.900753 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" event={"ID":"e8d201df-b678-414a-92c6-38192347216d","Type":"ContainerDied","Data":"c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd"} Nov 03 21:17:27.900896 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.900806 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k" event={"ID":"e8d201df-b678-414a-92c6-38192347216d","Type":"ContainerDied","Data":"ca245840e4200f49f3c8aeee9c98d312526dbec4392e31a9197b09acbb3a9dce"} Nov 03 21:17:27.911664 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.911642 5932 scope.go:122] "RemoveContainer" containerID="21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c" Nov 03 21:17:27.911919 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:27.911895 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c\": container with ID starting with 21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c not found: ID does not exist" containerID="21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c" Nov 03 21:17:27.911992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.911928 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c"} err="failed to get container status \"21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c\": rpc error: code = NotFound desc = could not find container \"21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c\": container with ID starting with 21fc73a363174fa40e791a1f4f78f3a6d2b47dad14746ce676014b4635371a3c not found: ID does not exist" Nov 03 21:17:27.911992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.911944 5932 scope.go:122] "RemoveContainer" containerID="c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd" Nov 03 21:17:27.920577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.920557 5932 scope.go:122] "RemoveContainer" containerID="c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd" Nov 03 21:17:27.920870 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:27.920852 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd\": container with ID starting with c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd not found: ID does not exist" containerID="c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd" Nov 03 21:17:27.920912 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:27.920878 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd"} err="failed to get container status \"c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd\": rpc error: code = NotFound desc = could not find container \"c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd\": container with ID starting with c5e8a72d434be029e809087db635d5bd99cf7e4676043857f706797a4e0b5bbd not found: ID does not exist" Nov 03 21:17:28.054030 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.053944 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp"] Nov 03 21:17:28.057779 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.057753 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-0cd49b-7b5db76459-k22kp"] Nov 03 21:17:28.159923 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.159875 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k"] Nov 03 21:17:28.164655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.164613 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-0cd49b-66dd9b96d5-kwh4k"] Nov 03 21:17:28.593888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.593852 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg"] Nov 03 21:17:28.594313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594299 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" containerName="revoke-keys" Nov 03 21:17:28.594356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594313 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" containerName="revoke-keys" Nov 03 21:17:28.594356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594331 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d2d6912c-89d8-414e-ba8c-d3bec81b13b1" containerName="payload-pre-processing" Nov 03 21:17:28.594356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594336 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d6912c-89d8-414e-ba8c-d3bec81b13b1" containerName="payload-pre-processing" Nov 03 21:17:28.594356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594356 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594361 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594374 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594380 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594388 5932 cpu_manager.go:414] "containerMap: removing container" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594393 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594399 5932 cpu_manager.go:414] "containerMap: removing container" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594404 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594415 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e8d201df-b678-414a-92c6-38192347216d" containerName="payload-processing" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594420 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d201df-b678-414a-92c6-38192347216d" containerName="payload-processing" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594430 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" Nov 03 21:17:28.594493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594435 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594504 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c454362-f7e1-4ef6-94dd-fb29158f12b9" containerName="istio-proxy" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594514 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" containerName="revoke-keys" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594521 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d6912c-89d8-414e-ba8c-d3bec81b13b1" containerName="payload-pre-processing" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594529 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594537 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1798e6-5adf-4c58-9409-869c2ffd2d38" containerName="istio-proxy" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594543 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e55545e-c3ca-4550-86f8-ba0b7b12c513" containerName="istio-proxy" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594549 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="393d8b8c-f4e6-4e66-b329-8cad52013719" containerName="istio-proxy" Nov 03 21:17:28.594864 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.594555 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d201df-b678-414a-92c6-38192347216d" containerName="payload-processing" Nov 03 21:17:28.599289 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.599265 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.650953 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.650923 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:17:28.659103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.659073 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg"] Nov 03 21:17:28.661919 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.661896 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv"] Nov 03 21:17:28.662650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.662610 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" Nov 03 21:17:28.662650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.662649 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" Nov 03 21:17:28.662827 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.662795 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab015e-f0c9-4936-8734-d1964c9cd4f1" containerName="istio-proxy" Nov 03 21:17:28.666018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.665997 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.694986 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.694960 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8c22\" (UniqueName: \"kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.695119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.695003 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.695179 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.695120 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.716429 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.716401 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv"] Nov 03 21:17:28.751827 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.751795 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:17:28.796081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.796048 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.796255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.796091 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tv49\" (UniqueName: \"kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.796255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.796209 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8c22\" (UniqueName: \"kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.796403 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.796256 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.796403 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.796315 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.796403 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.796352 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.797036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.797011 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.798699 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.798675 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.854041 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.853977 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8c22\" (UniqueName: \"kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22\") pod \"maas-api-revoke-sub-a8985ac3-kwdgg\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.897081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.897041 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.897244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.897098 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tv49\" (UniqueName: \"kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.897331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.897247 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.897924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.897900 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.899494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.899473 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.913792 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.913755 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:28.955117 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.955026 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tv49\" (UniqueName: \"kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49\") pod \"maas-api-revoke-sub-0137c46e-49mwv\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:28.980793 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:28.980514 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:29.287444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:29.287412 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg"] Nov 03 21:17:29.292946 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:17:29.292916 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4408b4_fe22_4c79_9b42_f12eac2292ac.slice/crio-66319ea1fd5e630752f77108201686e611cfd9acdd5ef7c97ffe894cd7da7bab WatchSource:0}: Error finding container 66319ea1fd5e630752f77108201686e611cfd9acdd5ef7c97ffe894cd7da7bab: Status 404 returned error can't find the container with id 66319ea1fd5e630752f77108201686e611cfd9acdd5ef7c97ffe894cd7da7bab Nov 03 21:17:29.389328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:29.389297 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv"] Nov 03 21:17:29.394598 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:17:29.394556 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2aae20_9e84_42a3_af2a_67a72c54068e.slice/crio-0f30fbc5d6823a799dd8dda0eb1d67fcdff1bc26cb97bde1a2b6200ffe700a57 WatchSource:0}: Error finding container 0f30fbc5d6823a799dd8dda0eb1d67fcdff1bc26cb97bde1a2b6200ffe700a57: Status 404 returned error can't find the container with id 0f30fbc5d6823a799dd8dda0eb1d67fcdff1bc26cb97bde1a2b6200ffe700a57 Nov 03 21:17:29.926206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:29.926167 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" event={"ID":"de4408b4-fe22-4c79-9b42-f12eac2292ac","Type":"ContainerStarted","Data":"fb80879a872fceaabcb2b533042bd8157f95921d8a714bf71775051572c8461b"} Nov 03 21:17:29.926645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:29.926219 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" event={"ID":"de4408b4-fe22-4c79-9b42-f12eac2292ac","Type":"ContainerStarted","Data":"66319ea1fd5e630752f77108201686e611cfd9acdd5ef7c97ffe894cd7da7bab"} Nov 03 21:17:29.927984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:29.927955 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" event={"ID":"2d2aae20-9e84-42a3-af2a-67a72c54068e","Type":"ContainerStarted","Data":"45c279828300bf5c4842f107539455bb4f6e1d622d7f2e27e95104390ce50872"} Nov 03 21:17:29.928097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:29.927999 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" event={"ID":"2d2aae20-9e84-42a3-af2a-67a72c54068e","Type":"ContainerStarted","Data":"0f30fbc5d6823a799dd8dda0eb1d67fcdff1bc26cb97bde1a2b6200ffe700a57"} Nov 03 21:17:30.035347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.035285 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" podStartSLOduration=2.035269366 podStartE2EDuration="2.035269366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:30.033475533 +0000 UTC m=+2858.098717005" watchObservedRunningTime="2026-11-03 21:17:30.035269366 +0000 UTC m=+2858.100510839" Nov 03 21:17:30.051252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.051216 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d6912c-89d8-414e-ba8c-d3bec81b13b1" path="/var/lib/kubelet/pods/d2d6912c-89d8-414e-ba8c-d3bec81b13b1/volumes" Nov 03 21:17:30.051642 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.051613 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d201df-b678-414a-92c6-38192347216d" path="/var/lib/kubelet/pods/e8d201df-b678-414a-92c6-38192347216d/volumes" Nov 03 21:17:30.142707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.142648 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" podStartSLOduration=2.142617574 podStartE2EDuration="2.142617574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:17:30.141001329 +0000 UTC m=+2858.206242803" watchObservedRunningTime="2026-11-03 21:17:30.142617574 +0000 UTC m=+2858.207859048" Nov 03 21:17:30.183216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.183125 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" containerID="cri-o://897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1" gracePeriod=45 Nov 03 21:17:30.189810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.189775 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" containerID="cri-o://f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7" gracePeriod=45 Nov 03 21:17:30.206072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.206037 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" containerID="cri-o://a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663" gracePeriod=45 Nov 03 21:17:30.209678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.209650 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": EOF" Nov 03 21:17:30.301842 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.301806 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.114:8000/ready\": dial tcp 10.132.0.114:8000: connect: connection refused" Nov 03 21:17:30.489273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.489242 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 21:17:30.519161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.519135 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:17:30.524078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.524057 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:17:30.623084 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623048 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm\") pod \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " Nov 03 21:17:30.623264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623106 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir\") pod \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " Nov 03 21:17:30.623264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623202 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home\") pod \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " Nov 03 21:17:30.623264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623243 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs\") pod \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " Nov 03 21:17:30.623454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623277 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh\" (UniqueName: \"kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh\") pod \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " Nov 03 21:17:30.623454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623315 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home\") pod \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " Nov 03 21:17:30.623454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623361 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm\") pod \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " Nov 03 21:17:30.623454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623382 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir" pod "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" (UID: "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.623454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623395 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home\") pod \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " Nov 03 21:17:30.623454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623432 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6\" (UniqueName: \"kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6\") pod \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623479 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir\") pod \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623519 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache\") pod \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623561 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache\") pod \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623659 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home" pod "692a7c2e-4cbf-407c-b821-aeb0582d9e32" (UID: "692a7c2e-4cbf-407c-b821-aeb0582d9e32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623673 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir\") pod \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623741 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj\" (UniqueName: \"kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj\") pod \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " Nov 03 21:17:30.623833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623779 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache\") pod \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\" (UID: \"0a379423-fe9d-4b10-b62a-1a9b5442cd4a\") " Nov 03 21:17:30.624219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623848 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs\") pod \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " Nov 03 21:17:30.624219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623878 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs\" (UniqueName: \"kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs\") pod \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\" (UID: \"692a7c2e-4cbf-407c-b821-aeb0582d9e32\") " Nov 03 21:17:30.624219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623887 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir" pod "0a379423-fe9d-4b10-b62a-1a9b5442cd4a" (UID: "0a379423-fe9d-4b10-b62a-1a9b5442cd4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.624219 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.623937 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm\") pod \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\" (UID: \"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8\") " Nov 03 21:17:30.624460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624240 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home" pod "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" (UID: "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.624460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624452 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.624575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624472 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.624575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624487 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.624575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624501 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.624575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624526 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home" pod "0a379423-fe9d-4b10-b62a-1a9b5442cd4a" (UID: "0a379423-fe9d-4b10-b62a-1a9b5442cd4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.624839 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.624771 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache" pod "0a379423-fe9d-4b10-b62a-1a9b5442cd4a" (UID: "0a379423-fe9d-4b10-b62a-1a9b5442cd4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.626406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626374 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm" pod "692a7c2e-4cbf-407c-b821-aeb0582d9e32" (UID: "692a7c2e-4cbf-407c-b821-aeb0582d9e32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.626406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626392 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6" pod "692a7c2e-4cbf-407c-b821-aeb0582d9e32" (UID: "692a7c2e-4cbf-407c-b821-aeb0582d9e32"). InnerVolumeSpecName "kube-api-access-qrfp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:30.626636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626416 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm" pod "0a379423-fe9d-4b10-b62a-1a9b5442cd4a" (UID: "0a379423-fe9d-4b10-b62a-1a9b5442cd4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.626636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626450 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh" pod "0a379423-fe9d-4b10-b62a-1a9b5442cd4a" (UID: "0a379423-fe9d-4b10-b62a-1a9b5442cd4a"). InnerVolumeSpecName "kube-api-access-k6nsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:30.626746 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626670 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache" pod "692a7c2e-4cbf-407c-b821-aeb0582d9e32" (UID: "692a7c2e-4cbf-407c-b821-aeb0582d9e32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.626746 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626700 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache" pod "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" (UID: "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.626848 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.626744 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir" pod "692a7c2e-4cbf-407c-b821-aeb0582d9e32" (UID: "692a7c2e-4cbf-407c-b821-aeb0582d9e32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.628050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.628015 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm" pod "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" (UID: "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:17:30.628204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.628176 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs" pod "0a379423-fe9d-4b10-b62a-1a9b5442cd4a" (UID: "0a379423-fe9d-4b10-b62a-1a9b5442cd4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:30.628390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.628373 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs" pod "692a7c2e-4cbf-407c-b821-aeb0582d9e32" (UID: "692a7c2e-4cbf-407c-b821-aeb0582d9e32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:30.628843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.628822 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs" pod "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" (UID: "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:30.629103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.629076 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj" pod "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" (UID: "d41fe606-bbc7-4223-aa45-ab49d2b5fbc8"). InnerVolumeSpecName "kube-api-access-jmcmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:30.726220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726131 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726169 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726182 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726196 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726210 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6nsh\" (UniqueName: \"kubernetes.io/projected/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-kube-api-access-k6nsh\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726223 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726238 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrfp6\" (UniqueName: \"kubernetes.io/projected/692a7c2e-4cbf-407c-b821-aeb0582d9e32-kube-api-access-qrfp6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726252 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726265 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692a7c2e-4cbf-407c-b821-aeb0582d9e32-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726278 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726293 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmcmj\" (UniqueName: \"kubernetes.io/projected/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-kube-api-access-jmcmj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726306 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a379423-fe9d-4b10-b62a-1a9b5442cd4a-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726321 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.726559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.726333 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692a7c2e-4cbf-407c-b821-aeb0582d9e32-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:30.937866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.937832 5932 generic.go:356] "Generic (PLEG): container finished" podID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerID="897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1" exitCode=0 Nov 03 21:17:30.938367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.937880 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" Nov 03 21:17:30.938367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.937909 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" event={"ID":"692a7c2e-4cbf-407c-b821-aeb0582d9e32","Type":"ContainerDied","Data":"897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1"} Nov 03 21:17:30.938367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.937951 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf" event={"ID":"692a7c2e-4cbf-407c-b821-aeb0582d9e32","Type":"ContainerDied","Data":"3967f5c8911a3ad21c789b82f22244f6927fa93a26a519b9232f7e94ac6cf091"} Nov 03 21:17:30.938367 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.937968 5932 scope.go:122] "RemoveContainer" containerID="897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1" Nov 03 21:17:30.939676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.939648 5932 generic.go:356] "Generic (PLEG): container finished" podID="de4408b4-fe22-4c79-9b42-f12eac2292ac" containerID="fb80879a872fceaabcb2b533042bd8157f95921d8a714bf71775051572c8461b" exitCode=0 Nov 03 21:17:30.939795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.939740 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" event={"ID":"de4408b4-fe22-4c79-9b42-f12eac2292ac","Type":"ContainerDied","Data":"fb80879a872fceaabcb2b533042bd8157f95921d8a714bf71775051572c8461b"} Nov 03 21:17:30.941434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.941411 5932 generic.go:356] "Generic (PLEG): container finished" podID="2d2aae20-9e84-42a3-af2a-67a72c54068e" containerID="45c279828300bf5c4842f107539455bb4f6e1d622d7f2e27e95104390ce50872" exitCode=0 Nov 03 21:17:30.941522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.941492 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" event={"ID":"2d2aae20-9e84-42a3-af2a-67a72c54068e","Type":"ContainerDied","Data":"45c279828300bf5c4842f107539455bb4f6e1d622d7f2e27e95104390ce50872"} Nov 03 21:17:30.942931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.942908 5932 generic.go:356] "Generic (PLEG): container finished" podID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerID="a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663" exitCode=0 Nov 03 21:17:30.943082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.942970 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" Nov 03 21:17:30.943082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.943008 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" event={"ID":"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8","Type":"ContainerDied","Data":"a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663"} Nov 03 21:17:30.943082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.943043 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf" event={"ID":"d41fe606-bbc7-4223-aa45-ab49d2b5fbc8","Type":"ContainerDied","Data":"e5495b132549ee118e1c3a4ed37eed778537aec453bc08ac3223689704a01c39"} Nov 03 21:17:30.944869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.944846 5932 generic.go:356] "Generic (PLEG): container finished" podID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerID="f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7" exitCode=0 Nov 03 21:17:30.944977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.944873 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" event={"ID":"0a379423-fe9d-4b10-b62a-1a9b5442cd4a","Type":"ContainerDied","Data":"f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7"} Nov 03 21:17:30.944977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.944891 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" Nov 03 21:17:30.944977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.944907 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2" event={"ID":"0a379423-fe9d-4b10-b62a-1a9b5442cd4a","Type":"ContainerDied","Data":"70f78dabc482bc18a97f4c225df2f47c1e246c1ac30ba4d275d4fbd2f839fa41"} Nov 03 21:17:30.949819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.949718 5932 scope.go:122] "RemoveContainer" containerID="897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1" Nov 03 21:17:30.950026 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:30.950007 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1\": container with ID starting with 897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1 not found: ID does not exist" containerID="897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1" Nov 03 21:17:30.950108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.950035 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1"} err="failed to get container status \"897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1\": rpc error: code = NotFound desc = could not find container \"897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1\": container with ID starting with 897717c0a50cfaa4b06affa4590dd38f25a3b4433300bbfb29456dd133c6dda1 not found: ID does not exist" Nov 03 21:17:30.950108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.950057 5932 scope.go:122] "RemoveContainer" containerID="a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663" Nov 03 21:17:30.960908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.960890 5932 scope.go:122] "RemoveContainer" containerID="a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663" Nov 03 21:17:30.961204 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:30.961183 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663\": container with ID starting with a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663 not found: ID does not exist" containerID="a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663" Nov 03 21:17:30.961263 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.961213 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663"} err="failed to get container status \"a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663\": rpc error: code = NotFound desc = could not find container \"a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663\": container with ID starting with a508890db03252d2dcc3ea4e32d8fd67b5218df7f9d4859dbdf1be3f8a4df663 not found: ID does not exist" Nov 03 21:17:30.961263 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.961230 5932 scope.go:122] "RemoveContainer" containerID="f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7" Nov 03 21:17:30.970021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.970002 5932 scope.go:122] "RemoveContainer" containerID="f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7" Nov 03 21:17:30.970281 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:30.970266 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7\": container with ID starting with f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7 not found: ID does not exist" containerID="f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7" Nov 03 21:17:30.970330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:30.970289 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7"} err="failed to get container status \"f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7\": rpc error: code = NotFound desc = could not find container \"f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7\": container with ID starting with f62b8c4b783fab77ef37636bdb14373785aefadcc480449300f6842440b954f7 not found: ID does not exist" Nov 03 21:17:31.201552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:31.201520 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf"] Nov 03 21:17:31.204598 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:31.204569 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-2-w1-0cd49b-kserve-bdfb75649-lxczf"] Nov 03 21:17:31.305944 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:31.305909 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf"] Nov 03 21:17:31.309740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:31.309716 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/simulator-w1-0cd49b-kserve-68f4fd9f86-wbtmf"] Nov 03 21:17:31.525522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:31.525433 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2"] Nov 03 21:17:31.531459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:31.531427 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0cd49b/distinct-w1-0cd49b-kserve-75c74d77f4-nm6t2"] Nov 03 21:17:32.052457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.052424 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" path="/var/lib/kubelet/pods/0a379423-fe9d-4b10-b62a-1a9b5442cd4a/volumes" Nov 03 21:17:32.052942 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.052878 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" path="/var/lib/kubelet/pods/692a7c2e-4cbf-407c-b821-aeb0582d9e32/volumes" Nov 03 21:17:32.053293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.053274 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" path="/var/lib/kubelet/pods/d41fe606-bbc7-4223-aa45-ab49d2b5fbc8/volumes" Nov 03 21:17:32.104407 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.104380 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:32.136890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.136857 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:32.244168 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244087 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22\" (UniqueName: \"kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22\") pod \"de4408b4-fe22-4c79-9b42-f12eac2292ac\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " Nov 03 21:17:32.244168 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244144 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca\") pod \"de4408b4-fe22-4c79-9b42-f12eac2292ac\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " Nov 03 21:17:32.244418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244201 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49\" (UniqueName: \"kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49\") pod \"2d2aae20-9e84-42a3-af2a-67a72c54068e\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " Nov 03 21:17:32.244418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244301 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca\") pod \"de4408b4-fe22-4c79-9b42-f12eac2292ac\" (UID: \"de4408b4-fe22-4c79-9b42-f12eac2292ac\") " Nov 03 21:17:32.244418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244338 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca\") pod \"2d2aae20-9e84-42a3-af2a-67a72c54068e\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " Nov 03 21:17:32.244418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244392 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca\") pod \"2d2aae20-9e84-42a3-af2a-67a72c54068e\" (UID: \"2d2aae20-9e84-42a3-af2a-67a72c54068e\") " Nov 03 21:17:32.244869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.244838 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca" pod "de4408b4-fe22-4c79-9b42-f12eac2292ac" (UID: "de4408b4-fe22-4c79-9b42-f12eac2292ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:32.245082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.245056 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca" pod "2d2aae20-9e84-42a3-af2a-67a72c54068e" (UID: "2d2aae20-9e84-42a3-af2a-67a72c54068e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:17:32.246915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.246885 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca" pod "de4408b4-fe22-4c79-9b42-f12eac2292ac" (UID: "de4408b4-fe22-4c79-9b42-f12eac2292ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:32.247152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.247124 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22" pod "de4408b4-fe22-4c79-9b42-f12eac2292ac" (UID: "de4408b4-fe22-4c79-9b42-f12eac2292ac"). InnerVolumeSpecName "kube-api-access-f8c22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:32.247814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.247788 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca" pod "2d2aae20-9e84-42a3-af2a-67a72c54068e" (UID: "2d2aae20-9e84-42a3-af2a-67a72c54068e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:17:32.247921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.247850 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49" pod "2d2aae20-9e84-42a3-af2a-67a72c54068e" (UID: "2d2aae20-9e84-42a3-af2a-67a72c54068e"). InnerVolumeSpecName "kube-api-access-5tv49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:17:32.345296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.345260 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8c22\" (UniqueName: \"kubernetes.io/projected/de4408b4-fe22-4c79-9b42-f12eac2292ac-kube-api-access-f8c22\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:32.345296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.345293 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:32.345296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.345303 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tv49\" (UniqueName: \"kubernetes.io/projected/2d2aae20-9e84-42a3-af2a-67a72c54068e-kube-api-access-5tv49\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:32.345685 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.345312 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4408b4-fe22-4c79-9b42-f12eac2292ac-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:32.345685 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.345322 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:32.345685 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.345330 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d2aae20-9e84-42a3-af2a-67a72c54068e-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:17:32.961751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.961718 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" Nov 03 21:17:32.961751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.961724 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg" event={"ID":"de4408b4-fe22-4c79-9b42-f12eac2292ac","Type":"ContainerDied","Data":"66319ea1fd5e630752f77108201686e611cfd9acdd5ef7c97ffe894cd7da7bab"} Nov 03 21:17:32.961993 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.961809 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66319ea1fd5e630752f77108201686e611cfd9acdd5ef7c97ffe894cd7da7bab" Nov 03 21:17:32.963111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.963091 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" Nov 03 21:17:32.963218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.963133 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv" event={"ID":"2d2aae20-9e84-42a3-af2a-67a72c54068e","Type":"ContainerDied","Data":"0f30fbc5d6823a799dd8dda0eb1d67fcdff1bc26cb97bde1a2b6200ffe700a57"} Nov 03 21:17:32.963218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:32.963190 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f30fbc5d6823a799dd8dda0eb1d67fcdff1bc26cb97bde1a2b6200ffe700a57" Nov 03 21:17:44.150351 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:17:44.150312 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:17:55.987242 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:55.987205 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r"] Nov 03 21:17:56.034970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:56.034931 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-0cd49b-rk72r"] Nov 03 21:17:56.050947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:17:56.050915 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f" path="/var/lib/kubelet/pods/0fbfd091-d5cd-4e17-92f2-8fbcf3844e2f/volumes" Nov 03 21:18:00.660712 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:18:00.660503 5932 scope.go:122] "RemoveContainer" containerID="78239b3dc147093f50271862b537922a09b5b98364a16be45216bb824ad06f9a" Nov 03 21:18:00.672409 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:18:00.672381 5932 scope.go:122] "RemoveContainer" containerID="a5ab54e85c234628035acf5c320873eaa9fdfc132e1551151d11fc8f45027436" Nov 03 21:18:00.682708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:18:00.682681 5932 scope.go:122] "RemoveContainer" containerID="ca37cb9ae60f0428fa0da016b5435ca10955f970e2c5303780d6424f7ea74a1a" Nov 03 21:18:14.152352 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:18:14.152263 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:18:44.150656 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:18:44.150603 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:19:00.763460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:00.763427 5932 scope.go:122] "RemoveContainer" containerID="3dafd483c1171186c47102c8471dd4500c89db49d945bbe964b2c64db1dad245" Nov 03 21:19:00.773263 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:00.773240 5932 scope.go:122] "RemoveContainer" containerID="4a0483c9d568a61451c608231eef17886f65124c9f95db06b350730f6c160dcb" Nov 03 21:19:14.150963 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:19:14.150922 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:19:42.345278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345194 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r"] Nov 03 21:19:42.345688 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345676 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2d2aae20-9e84-42a3-af2a-67a72c54068e" containerName="revoke-keys" Nov 03 21:19:42.345737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345689 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2aae20-9e84-42a3-af2a-67a72c54068e" containerName="revoke-keys" Nov 03 21:19:42.345737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345720 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" Nov 03 21:19:42.345737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345725 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" Nov 03 21:19:42.345737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345734 5932 cpu_manager.go:414] "containerMap: removing container" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345739 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345747 5932 cpu_manager.go:414] "containerMap: removing container" podUID="de4408b4-fe22-4c79-9b42-f12eac2292ac" containerName="revoke-keys" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345751 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4408b4-fe22-4c79-9b42-f12eac2292ac" containerName="revoke-keys" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345768 5932 cpu_manager.go:414] "containerMap: removing container" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345772 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345833 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="d41fe606-bbc7-4223-aa45-ab49d2b5fbc8" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345844 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2aae20-9e84-42a3-af2a-67a72c54068e" containerName="revoke-keys" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345854 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a379423-fe9d-4b10-b62a-1a9b5442cd4a" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345859 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="692a7c2e-4cbf-407c-b821-aeb0582d9e32" containerName="main" Nov 03 21:19:42.345866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.345865 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4408b4-fe22-4c79-9b42-f12eac2292ac" containerName="revoke-keys" Nov 03 21:19:42.349104 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.349084 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.407911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.407865 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r"] Nov 03 21:19:42.426868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.426824 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gbh\" (UniqueName: \"kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.427061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.426886 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.427061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.427014 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.450003 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.449964 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:42.450223 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.450016 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:19:42.528494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.528452 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.528746 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.528541 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.528833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.528753 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gbh\" (UniqueName: \"kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.529298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.529273 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.531040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.531019 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.587011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.586971 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gbh\" (UniqueName: \"kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh\") pod \"maas-api-revoke-sub-3005de46-qzw7r\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:42.662802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:42.662762 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:43.037888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:43.037804 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r"] Nov 03 21:19:43.043893 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:19:43.043862 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9b483b_3187_4575_b56d_48d2e929d4dd.slice/crio-1965d7bd1a9086a3776542ceca8c79b88217cef69e5be1a980175055af978546 WatchSource:0}: Error finding container 1965d7bd1a9086a3776542ceca8c79b88217cef69e5be1a980175055af978546: Status 404 returned error can't find the container with id 1965d7bd1a9086a3776542ceca8c79b88217cef69e5be1a980175055af978546 Nov 03 21:19:43.802330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:43.802300 5932 generic.go:356] "Generic (PLEG): container finished" podID="0f9b483b-3187-4575-b56d-48d2e929d4dd" containerID="134314d64eca1c1f6b4f8ede4f2a38befd7e86d28aa2dc5e1aae337af907b565" exitCode=0 Nov 03 21:19:43.802776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:43.802377 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" event={"ID":"0f9b483b-3187-4575-b56d-48d2e929d4dd","Type":"ContainerDied","Data":"134314d64eca1c1f6b4f8ede4f2a38befd7e86d28aa2dc5e1aae337af907b565"} Nov 03 21:19:43.802776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:43.802472 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" event={"ID":"0f9b483b-3187-4575-b56d-48d2e929d4dd","Type":"ContainerStarted","Data":"1965d7bd1a9086a3776542ceca8c79b88217cef69e5be1a980175055af978546"} Nov 03 21:19:44.150451 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:19:44.150415 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:19:44.945109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:44.945086 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:45.054593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.054561 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh\" (UniqueName: \"kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh\") pod \"0f9b483b-3187-4575-b56d-48d2e929d4dd\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " Nov 03 21:19:45.054751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.054692 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca\") pod \"0f9b483b-3187-4575-b56d-48d2e929d4dd\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " Nov 03 21:19:45.054806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.054785 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca\") pod \"0f9b483b-3187-4575-b56d-48d2e929d4dd\" (UID: \"0f9b483b-3187-4575-b56d-48d2e929d4dd\") " Nov 03 21:19:45.055138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.055112 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca" pod "0f9b483b-3187-4575-b56d-48d2e929d4dd" (UID: "0f9b483b-3187-4575-b56d-48d2e929d4dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:19:45.056720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.056694 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca" pod "0f9b483b-3187-4575-b56d-48d2e929d4dd" (UID: "0f9b483b-3187-4575-b56d-48d2e929d4dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:19:45.056720 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.056713 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh" pod "0f9b483b-3187-4575-b56d-48d2e929d4dd" (UID: "0f9b483b-3187-4575-b56d-48d2e929d4dd"). InnerVolumeSpecName "kube-api-access-r4gbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:19:45.156368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.156331 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4gbh\" (UniqueName: \"kubernetes.io/projected/0f9b483b-3187-4575-b56d-48d2e929d4dd-kube-api-access-r4gbh\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:45.156368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.156357 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:45.156368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.156367 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9b483b-3187-4575-b56d-48d2e929d4dd-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:19:45.814932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.814903 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" Nov 03 21:19:45.814932 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.814929 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r" event={"ID":"0f9b483b-3187-4575-b56d-48d2e929d4dd","Type":"ContainerDied","Data":"1965d7bd1a9086a3776542ceca8c79b88217cef69e5be1a980175055af978546"} Nov 03 21:19:45.815253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:45.814972 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1965d7bd1a9086a3776542ceca8c79b88217cef69e5be1a980175055af978546" Nov 03 21:19:56.196957 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.196925 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:19:56.212335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.212308 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:19:56.409050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.409011 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt"] Nov 03 21:19:56.409563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.409543 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0f9b483b-3187-4575-b56d-48d2e929d4dd" containerName="revoke-keys" Nov 03 21:19:56.409671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.409565 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9b483b-3187-4575-b56d-48d2e929d4dd" containerName="revoke-keys" Nov 03 21:19:56.409671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.409658 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9b483b-3187-4575-b56d-48d2e929d4dd" containerName="revoke-keys" Nov 03 21:19:56.413111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.413092 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.464211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.464110 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:19:56.467853 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.467825 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt"] Nov 03 21:19:56.510278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.510248 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:19:56.567947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.567910 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.568136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.567980 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zllf\" (UniqueName: \"kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.568136 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.568073 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.668982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.668939 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.669216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.669034 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zllf\" (UniqueName: \"kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.669216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.669103 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.669795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.669773 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.671442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.671424 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:56.726707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:56.726639 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zllf\" (UniqueName: \"kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf\") pod \"maas-api-revoke-sub-6fac3c8c-bbwdt\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:57.026490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:57.026405 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:19:57.398570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:57.398530 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt"] Nov 03 21:19:57.405357 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:19:57.405312 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b26eeef_96df_4849_aeb4_654617e15fb0.slice/crio-95812dcfe7329ecef29dc55e8177a2a9d1200910e1dddd5fd1ec7463d0eb12cf WatchSource:0}: Error finding container 95812dcfe7329ecef29dc55e8177a2a9d1200910e1dddd5fd1ec7463d0eb12cf: Status 404 returned error can't find the container with id 95812dcfe7329ecef29dc55e8177a2a9d1200910e1dddd5fd1ec7463d0eb12cf Nov 03 21:19:57.891074 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:57.891039 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" event={"ID":"2b26eeef-96df-4849-aeb4-654617e15fb0","Type":"ContainerStarted","Data":"47b3d0be22e0992c112ee75a4afc63123adb91c088de42132a2be8095ed2a642"} Nov 03 21:19:57.891262 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:57.891094 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" event={"ID":"2b26eeef-96df-4849-aeb4-654617e15fb0","Type":"ContainerStarted","Data":"95812dcfe7329ecef29dc55e8177a2a9d1200910e1dddd5fd1ec7463d0eb12cf"} Nov 03 21:19:58.000148 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:58.000035 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" podStartSLOduration=2.000021775 podStartE2EDuration="2.000021775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:19:57.998792127 +0000 UTC m=+3006.064033599" watchObservedRunningTime="2026-11-03 21:19:58.000021775 +0000 UTC m=+3006.065263269" Nov 03 21:19:58.900165 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:58.900124 5932 generic.go:356] "Generic (PLEG): container finished" podID="2b26eeef-96df-4849-aeb4-654617e15fb0" containerID="47b3d0be22e0992c112ee75a4afc63123adb91c088de42132a2be8095ed2a642" exitCode=0 Nov 03 21:19:58.900592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:19:58.900206 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" event={"ID":"2b26eeef-96df-4849-aeb4-654617e15fb0","Type":"ContainerDied","Data":"47b3d0be22e0992c112ee75a4afc63123adb91c088de42132a2be8095ed2a642"} Nov 03 21:20:00.044817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.044788 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:20:00.093805 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.093770 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf\" (UniqueName: \"kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf\") pod \"2b26eeef-96df-4849-aeb4-654617e15fb0\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " Nov 03 21:20:00.093974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.093888 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca\") pod \"2b26eeef-96df-4849-aeb4-654617e15fb0\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " Nov 03 21:20:00.093974 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.093911 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca\") pod \"2b26eeef-96df-4849-aeb4-654617e15fb0\" (UID: \"2b26eeef-96df-4849-aeb4-654617e15fb0\") " Nov 03 21:20:00.094401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.094368 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca" pod "2b26eeef-96df-4849-aeb4-654617e15fb0" (UID: "2b26eeef-96df-4849-aeb4-654617e15fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:00.096100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.096073 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca" pod "2b26eeef-96df-4849-aeb4-654617e15fb0" (UID: "2b26eeef-96df-4849-aeb4-654617e15fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:00.096244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.096097 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf" pod "2b26eeef-96df-4849-aeb4-654617e15fb0" (UID: "2b26eeef-96df-4849-aeb4-654617e15fb0"). InnerVolumeSpecName "kube-api-access-2zllf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:00.195016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.194926 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zllf\" (UniqueName: \"kubernetes.io/projected/2b26eeef-96df-4849-aeb4-654617e15fb0-kube-api-access-2zllf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:00.195016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.194957 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:00.195016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.194967 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b26eeef-96df-4849-aeb4-654617e15fb0-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:00.800420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.800390 5932 scope.go:122] "RemoveContainer" containerID="e0da914476a95fa295f360017fae838983ba9fc142d72b23a3390cd11d8092ed" Nov 03 21:20:00.912832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.912801 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" Nov 03 21:20:00.912988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.912836 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt" event={"ID":"2b26eeef-96df-4849-aeb4-654617e15fb0","Type":"ContainerDied","Data":"95812dcfe7329ecef29dc55e8177a2a9d1200910e1dddd5fd1ec7463d0eb12cf"} Nov 03 21:20:00.912988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:00.912882 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95812dcfe7329ecef29dc55e8177a2a9d1200910e1dddd5fd1ec7463d0eb12cf" Nov 03 21:20:14.150310 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:20:14.150268 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:20:17.613726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.613691 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn"] Nov 03 21:20:17.614448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.614396 5932 cpu_manager.go:414] "containerMap: removing container" podUID="2b26eeef-96df-4849-aeb4-654617e15fb0" containerName="revoke-keys" Nov 03 21:20:17.614448 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.614417 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b26eeef-96df-4849-aeb4-654617e15fb0" containerName="revoke-keys" Nov 03 21:20:17.614653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.614577 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b26eeef-96df-4849-aeb4-654617e15fb0" containerName="revoke-keys" Nov 03 21:20:17.619302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.619281 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.665822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.665789 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.665970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.665858 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwq7\" (UniqueName: \"kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.665970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.665888 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.668735 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.668696 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:20:17.672463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.672436 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn"] Nov 03 21:20:17.714970 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.714947 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:20:17.766763 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.766727 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.766950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.766804 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwq7\" (UniqueName: \"kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.766950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.766834 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.767493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.767471 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.769205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.769176 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.825165 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.825135 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwq7\" (UniqueName: \"kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7\") pod \"maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:17.932238 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:17.932207 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:18.305251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:18.305168 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn"] Nov 03 21:20:18.311591 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:20:18.311563 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce67fd0_fc81_4565_9f66_07d5c72779e3.slice/crio-469ca1e426577f410cddfd424f3f744a49c33794dc0a6c8df33a68373953e865 WatchSource:0}: Error finding container 469ca1e426577f410cddfd424f3f744a49c33794dc0a6c8df33a68373953e865: Status 404 returned error can't find the container with id 469ca1e426577f410cddfd424f3f744a49c33794dc0a6c8df33a68373953e865 Nov 03 21:20:19.026507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:19.026405 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" event={"ID":"0ce67fd0-fc81-4565-9f66-07d5c72779e3","Type":"ContainerStarted","Data":"4184e9a23bcf3ed5b2520d56c4a4f6d4fbdb4ed04113410dd591f3f66433a3b5"} Nov 03 21:20:19.026507 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:19.026494 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" event={"ID":"0ce67fd0-fc81-4565-9f66-07d5c72779e3","Type":"ContainerStarted","Data":"469ca1e426577f410cddfd424f3f744a49c33794dc0a6c8df33a68373953e865"} Nov 03 21:20:19.136121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:19.136061 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" podStartSLOduration=2.136047061 podStartE2EDuration="2.136047061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:20:19.133847437 +0000 UTC m=+3027.199088914" watchObservedRunningTime="2026-11-03 21:20:19.136047061 +0000 UTC m=+3027.201288547" Nov 03 21:20:20.036045 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:20.036007 5932 generic.go:356] "Generic (PLEG): container finished" podID="0ce67fd0-fc81-4565-9f66-07d5c72779e3" containerID="4184e9a23bcf3ed5b2520d56c4a4f6d4fbdb4ed04113410dd591f3f66433a3b5" exitCode=0 Nov 03 21:20:20.036442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:20.036089 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" event={"ID":"0ce67fd0-fc81-4565-9f66-07d5c72779e3","Type":"ContainerDied","Data":"4184e9a23bcf3ed5b2520d56c4a4f6d4fbdb4ed04113410dd591f3f66433a3b5"} Nov 03 21:20:21.191443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.191417 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:21.303604 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.303512 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca\") pod \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " Nov 03 21:20:21.303604 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.303563 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca\") pod \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " Nov 03 21:20:21.303892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.303869 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7\" (UniqueName: \"kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7\") pod \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\" (UID: \"0ce67fd0-fc81-4565-9f66-07d5c72779e3\") " Nov 03 21:20:21.303954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.303934 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca" pod "0ce67fd0-fc81-4565-9f66-07d5c72779e3" (UID: "0ce67fd0-fc81-4565-9f66-07d5c72779e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:21.304416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.304389 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:21.306319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.306287 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca" pod "0ce67fd0-fc81-4565-9f66-07d5c72779e3" (UID: "0ce67fd0-fc81-4565-9f66-07d5c72779e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:20:21.306533 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.306513 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7" pod "0ce67fd0-fc81-4565-9f66-07d5c72779e3" (UID: "0ce67fd0-fc81-4565-9f66-07d5c72779e3"). InnerVolumeSpecName "kube-api-access-hdwq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:21.405924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.405892 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ce67fd0-fc81-4565-9f66-07d5c72779e3-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:21.405924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:21.405920 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdwq7\" (UniqueName: \"kubernetes.io/projected/0ce67fd0-fc81-4565-9f66-07d5c72779e3-kube-api-access-hdwq7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:22.051185 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:22.051146 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" event={"ID":"0ce67fd0-fc81-4565-9f66-07d5c72779e3","Type":"ContainerDied","Data":"469ca1e426577f410cddfd424f3f744a49c33794dc0a6c8df33a68373953e865"} Nov 03 21:20:22.051368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:22.051205 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="469ca1e426577f410cddfd424f3f744a49c33794dc0a6c8df33a68373953e865" Nov 03 21:20:22.051368 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:22.051168 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn" Nov 03 21:20:28.050882 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:28.050852 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc"] Nov 03 21:20:28.092693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:28.092664 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0dba083a-bhvsc"] Nov 03 21:20:30.051814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:30.051777 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f654c0fc-d13f-49a9-aee6-7470756118a2" path="/var/lib/kubelet/pods/f654c0fc-d13f-49a9-aee6-7470756118a2/volumes" Nov 03 21:20:36.738401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:36.738360 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn"] Nov 03 21:20:36.787282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:36.787243 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f1855-bqcmn"] Nov 03 21:20:38.051231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:38.051193 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce67fd0-fc81-4565-9f66-07d5c72779e3" path="/var/lib/kubelet/pods/0ce67fd0-fc81-4565-9f66-07d5c72779e3/volumes" Nov 03 21:20:42.053364 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:42.053334 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc"] Nov 03 21:20:42.094655 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:42.094586 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b23bde-qg5dc"] Nov 03 21:20:43.779424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:43.779392 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm"] Nov 03 21:20:43.779887 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:43.779594 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" containerID="cri-o://080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf" gracePeriod=30 Nov 03 21:20:43.793605 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:43.793577 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc"] Nov 03 21:20:43.793794 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:43.793773 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" containerID="cri-o://f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f" gracePeriod=30 Nov 03 21:20:44.050748 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:44.050664 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a37e9d4-d5db-4538-8b63-53e1da441f51" path="/var/lib/kubelet/pods/9a37e9d4-d5db-4538-8b63-53e1da441f51/volumes" Nov 03 21:20:44.149924 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:20:44.149888 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:20:49.051980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.051914 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:20:49.082965 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.082942 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:20:49.090068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090047 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090149 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090133 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090191 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090168 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090240 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090225 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090260 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090296 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090381 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090352 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090418 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090503 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090483 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket\") pod \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\" (UID: \"19d01288-f79b-44ca-8aa6-ebad2ce91abe\") " Nov 03 21:20:49.090565 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090511 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.090657 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090604 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:49.090769 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090749 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.090963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090941 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.090963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090947 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.091118 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090960 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.091118 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.090967 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.091118 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.091000 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.092384 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.092361 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:20:49.092464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.092368 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:49.092464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.092422 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "kube-api-access-6wvrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:49.092543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.092482 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy" pod "19d01288-f79b-44ca-8aa6-ebad2ce91abe" (UID: "19d01288-f79b-44ca-8aa6-ebad2ce91abe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.191516 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191479 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.191516 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191521 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.191765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191564 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.191765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191597 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.191765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191667 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.191765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191695 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.191765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191739 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.192046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191768 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.192046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191843 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4\") pod \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\" (UID: \"f23e80b5-7792-4aa0-8ee3-ca98751d70aa\") " Nov 03 21:20:49.192046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.191944 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.192207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192048 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:20:49.192207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192090 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.192303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192205 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.192303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192224 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.192411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192384 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192397 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192406 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192415 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192425 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvrj\" (UniqueName: \"kubernetes.io/projected/19d01288-f79b-44ca-8aa6-ebad2ce91abe-kube-api-access-6wvrj\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192435 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192444 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192465 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192484 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192498 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.192574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.192513 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19d01288-f79b-44ca-8aa6-ebad2ce91abe-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.194290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.194265 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:49.194414 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.194321 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:20:49.194537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.194514 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "kube-api-access-7cgz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:20:49.194537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.194634 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy" pod "f23e80b5-7792-4aa0-8ee3-ca98751d70aa" (UID: "f23e80b5-7792-4aa0-8ee3-ca98751d70aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:20:49.244159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.244116 5932 generic.go:356] "Generic (PLEG): container finished" podID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerID="080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf" exitCode=0 Nov 03 21:20:49.244383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.244202 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" Nov 03 21:20:49.244383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.244221 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" event={"ID":"19d01288-f79b-44ca-8aa6-ebad2ce91abe","Type":"ContainerDied","Data":"080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf"} Nov 03 21:20:49.244383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.244297 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm" event={"ID":"19d01288-f79b-44ca-8aa6-ebad2ce91abe","Type":"ContainerDied","Data":"068e455a4d7b18f3259fa296ece40e8fcfe3c657bd475d60f94deebd71899e40"} Nov 03 21:20:49.244383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.244319 5932 scope.go:122] "RemoveContainer" containerID="080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf" Nov 03 21:20:49.255021 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.254984 5932 generic.go:356] "Generic (PLEG): container finished" podID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerID="f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f" exitCode=0 Nov 03 21:20:49.255159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.255041 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" Nov 03 21:20:49.255159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.255073 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" event={"ID":"f23e80b5-7792-4aa0-8ee3-ca98751d70aa","Type":"ContainerDied","Data":"f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f"} Nov 03 21:20:49.255159 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.255140 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc" event={"ID":"f23e80b5-7792-4aa0-8ee3-ca98751d70aa","Type":"ContainerDied","Data":"e7e9fece5d3680ebacc3a2cf6a0941a1d1e288e98485e30f144415d7996ef98b"} Nov 03 21:20:49.262941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.262915 5932 scope.go:122] "RemoveContainer" containerID="16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f" Nov 03 21:20:49.272135 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.272113 5932 scope.go:122] "RemoveContainer" containerID="080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf" Nov 03 21:20:49.272444 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:20:49.272425 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf\": container with ID starting with 080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf not found: ID does not exist" containerID="080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf" Nov 03 21:20:49.272495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.272453 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf"} err="failed to get container status \"080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf\": rpc error: code = NotFound desc = could not find container \"080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf\": container with ID starting with 080a379514a57fc13fb307ebda2352a5b5d9d0c6973c2dc14a3acc7a8ad507cf not found: ID does not exist" Nov 03 21:20:49.272495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.272469 5932 scope.go:122] "RemoveContainer" containerID="16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f" Nov 03 21:20:49.272798 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:20:49.272777 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f\": container with ID starting with 16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f not found: ID does not exist" containerID="16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f" Nov 03 21:20:49.272900 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.272807 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f"} err="failed to get container status \"16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f\": rpc error: code = NotFound desc = could not find container \"16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f\": container with ID starting with 16d75859d3b1e1c14ea6bf44d2ae2bdb9733e42277547799e112708e6a16c55f not found: ID does not exist" Nov 03 21:20:49.272900 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.272829 5932 scope.go:122] "RemoveContainer" containerID="f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f" Nov 03 21:20:49.282306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.282288 5932 scope.go:122] "RemoveContainer" containerID="f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f" Nov 03 21:20:49.282583 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:20:49.282562 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f\": container with ID starting with f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f not found: ID does not exist" containerID="f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f" Nov 03 21:20:49.282687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.282588 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f"} err="failed to get container status \"f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f\": rpc error: code = NotFound desc = could not find container \"f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f\": container with ID starting with f861adee3db005637b3ab6a9c26c65ec7cea00ca14ba4efa4a40294f3b89f97f not found: ID does not exist" Nov 03 21:20:49.293122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.293099 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.293122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.293123 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.293258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.293133 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cgz4\" (UniqueName: \"kubernetes.io/projected/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-kube-api-access-7cgz4\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.293258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.293143 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f23e80b5-7792-4aa0-8ee3-ca98751d70aa-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:20:49.429706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.429673 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc"] Nov 03 21:20:49.435994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.435963 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-mgnhc"] Nov 03 21:20:49.538342 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.538305 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm"] Nov 03 21:20:49.542707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:49.542677 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f1855-openshift-default-6b9cd5c6d7-dksvm"] Nov 03 21:20:50.051310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:50.051273 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" path="/var/lib/kubelet/pods/19d01288-f79b-44ca-8aa6-ebad2ce91abe/volumes" Nov 03 21:20:50.051799 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:20:50.051785 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" path="/var/lib/kubelet/pods/f23e80b5-7792-4aa0-8ee3-ca98751d70aa/volumes" Nov 03 21:21:00.825365 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:00.825329 5932 scope.go:122] "RemoveContainer" containerID="52153a27ad56a90530cfc1f0af602d0631f17d35c27e7cbb276910501df4120c" Nov 03 21:21:06.182852 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.182766 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp"] Nov 03 21:21:06.183250 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.182988 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" podUID="378e125d-9e59-412d-9337-9c29b73b5d9d" containerName="maas-api" containerID="cri-o://b96209bba11551fcbb44179b64e60beb092dd625505c8b7e8582d8b4bbd5a1fe" gracePeriod=30 Nov 03 21:21:06.370990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.370960 5932 generic.go:356] "Generic (PLEG): container finished" podID="378e125d-9e59-412d-9337-9c29b73b5d9d" containerID="b96209bba11551fcbb44179b64e60beb092dd625505c8b7e8582d8b4bbd5a1fe" exitCode=0 Nov 03 21:21:06.371123 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.371028 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" event={"ID":"378e125d-9e59-412d-9337-9c29b73b5d9d","Type":"ContainerDied","Data":"b96209bba11551fcbb44179b64e60beb092dd625505c8b7e8582d8b4bbd5a1fe"} Nov 03 21:21:06.446647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.446549 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.564008 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\") pod \"378e125d-9e59-412d-9337-9c29b73b5d9d\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.564598 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs\") pod \"378e125d-9e59-412d-9337-9c29b73b5d9d\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.564658 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr\" (UniqueName: \"kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr\") pod \"378e125d-9e59-412d-9337-9c29b73b5d9d\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.564765 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca\") pod \"378e125d-9e59-412d-9337-9c29b73b5d9d\" (UID: \"378e125d-9e59-412d-9337-9c29b73b5d9d\") " Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.565444 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca" pod "378e125d-9e59-412d-9337-9c29b73b5d9d" (UID: "378e125d-9e59-412d-9337-9c29b73b5d9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.576228 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr" pod "378e125d-9e59-412d-9337-9c29b73b5d9d" (UID: "378e125d-9e59-412d-9337-9c29b73b5d9d"). InnerVolumeSpecName "kube-api-access-77vdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.576902 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls" pod "378e125d-9e59-412d-9337-9c29b73b5d9d" (UID: "378e125d-9e59-412d-9337-9c29b73b5d9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:06.564415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.577000 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs" pod "378e125d-9e59-412d-9337-9c29b73b5d9d" (UID: "378e125d-9e59-412d-9337-9c29b73b5d9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:06.666439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.666396 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/378e125d-9e59-412d-9337-9c29b73b5d9d-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:06.666439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.666431 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:06.666439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.666444 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/378e125d-9e59-412d-9337-9c29b73b5d9d-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:06.666756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:06.666456 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77vdr\" (UniqueName: \"kubernetes.io/projected/378e125d-9e59-412d-9337-9c29b73b5d9d-kube-api-access-77vdr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:07.379890 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:07.379861 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" Nov 03 21:21:07.380354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:07.379902 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp" event={"ID":"378e125d-9e59-412d-9337-9c29b73b5d9d","Type":"ContainerDied","Data":"993a95c305a84937a1d05033d37e3b893e13ba6045fe0b58e21446cc5b170221"} Nov 03 21:21:07.380354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:07.379970 5932 scope.go:122] "RemoveContainer" containerID="b96209bba11551fcbb44179b64e60beb092dd625505c8b7e8582d8b4bbd5a1fe" Nov 03 21:21:07.539551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:07.539514 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp"] Nov 03 21:21:07.585602 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:07.585559 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c971dc-74cfd89786-j26xp"] Nov 03 21:21:08.050264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:08.050230 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="378e125d-9e59-412d-9337-9c29b73b5d9d" path="/var/lib/kubelet/pods/378e125d-9e59-412d-9337-9c29b73b5d9d/volumes" Nov 03 21:21:14.050827 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:14.050799 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7"] Nov 03 21:21:14.094227 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:14.094190 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bee2d2ed-bvtc7"] Nov 03 21:21:14.150731 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:21:14.150690 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:21:16.050231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:16.050189 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19753c32-cfc3-4e4b-946c-ea7c50472ac4" path="/var/lib/kubelet/pods/19753c32-cfc3-4e4b-946c-ea7c50472ac4/volumes" Nov 03 21:21:26.441339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441303 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v"] Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441726 5932 cpu_manager.go:414] "containerMap: removing container" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441738 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441762 5932 cpu_manager.go:414] "containerMap: removing container" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441768 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441778 5932 cpu_manager.go:414] "containerMap: removing container" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441783 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441795 5932 cpu_manager.go:414] "containerMap: removing container" podUID="378e125d-9e59-412d-9337-9c29b73b5d9d" containerName="maas-api" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441799 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="378e125d-9e59-412d-9337-9c29b73b5d9d" containerName="maas-api" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441816 5932 cpu_manager.go:414] "containerMap: removing container" podUID="0ce67fd0-fc81-4565-9f66-07d5c72779e3" containerName="revoke-keys" Nov 03 21:21:26.441873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441820 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce67fd0-fc81-4565-9f66-07d5c72779e3" containerName="revoke-keys" Nov 03 21:21:26.442206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441883 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" Nov 03 21:21:26.442206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441893 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d01288-f79b-44ca-8aa6-ebad2ce91abe" containerName="istio-proxy" Nov 03 21:21:26.442206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441898 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="378e125d-9e59-412d-9337-9c29b73b5d9d" containerName="maas-api" Nov 03 21:21:26.442206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441908 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce67fd0-fc81-4565-9f66-07d5c72779e3" containerName="revoke-keys" Nov 03 21:21:26.442206 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.441914 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23e80b5-7792-4aa0-8ee3-ca98751d70aa" containerName="istio-proxy" Nov 03 21:21:26.445274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.445257 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.465607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.465583 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.465735 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.465693 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.465790 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.465764 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qgr\" (UniqueName: \"kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.504967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.504947 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:21:26.550373 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.550336 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:21:26.552739 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.552714 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v"] Nov 03 21:21:26.566288 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.566258 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qgr\" (UniqueName: \"kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.566411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.566314 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.566542 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.566476 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.567144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.567125 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.568765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.568743 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.623927 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.623893 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qgr\" (UniqueName: \"kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr\") pod \"maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:26.761120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:26.761027 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:27.136287 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:27.136255 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v"] Nov 03 21:21:27.143546 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:21:27.143518 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebf4cec_4ed7_433e_8098_449d7dbc20a6.slice/crio-124187358f2c26fcefca7eeafc7b80ef117dabb7d816bb87a1e02b5e9fdf6b31 WatchSource:0}: Error finding container 124187358f2c26fcefca7eeafc7b80ef117dabb7d816bb87a1e02b5e9fdf6b31: Status 404 returned error can't find the container with id 124187358f2c26fcefca7eeafc7b80ef117dabb7d816bb87a1e02b5e9fdf6b31 Nov 03 21:21:27.503992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:27.503950 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" event={"ID":"debf4cec-4ed7-433e-8098-449d7dbc20a6","Type":"ContainerStarted","Data":"ac7ed39bf02573d6d5412422884e6b9a08279c66161b61e0f0582200223414bc"} Nov 03 21:21:27.504385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:27.504047 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" event={"ID":"debf4cec-4ed7-433e-8098-449d7dbc20a6","Type":"ContainerStarted","Data":"124187358f2c26fcefca7eeafc7b80ef117dabb7d816bb87a1e02b5e9fdf6b31"} Nov 03 21:21:27.618858 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:27.618804 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" podStartSLOduration=1.618789936 podStartE2EDuration="1.618789936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:27.61748054 +0000 UTC m=+3095.682722012" watchObservedRunningTime="2026-11-03 21:21:27.618789936 +0000 UTC m=+3095.684031442" Nov 03 21:21:28.513182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:28.513146 5932 generic.go:356] "Generic (PLEG): container finished" podID="debf4cec-4ed7-433e-8098-449d7dbc20a6" containerID="ac7ed39bf02573d6d5412422884e6b9a08279c66161b61e0f0582200223414bc" exitCode=0 Nov 03 21:21:28.513593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:28.513228 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" event={"ID":"debf4cec-4ed7-433e-8098-449d7dbc20a6","Type":"ContainerDied","Data":"ac7ed39bf02573d6d5412422884e6b9a08279c66161b61e0f0582200223414bc"} Nov 03 21:21:29.658405 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.658378 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:29.690425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.690396 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca\") pod \"debf4cec-4ed7-433e-8098-449d7dbc20a6\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " Nov 03 21:21:29.690567 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.690497 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca\") pod \"debf4cec-4ed7-433e-8098-449d7dbc20a6\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " Nov 03 21:21:29.690567 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.690521 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr\" (UniqueName: \"kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr\") pod \"debf4cec-4ed7-433e-8098-449d7dbc20a6\" (UID: \"debf4cec-4ed7-433e-8098-449d7dbc20a6\") " Nov 03 21:21:29.690795 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.690772 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca" pod "debf4cec-4ed7-433e-8098-449d7dbc20a6" (UID: "debf4cec-4ed7-433e-8098-449d7dbc20a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:29.692640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.692610 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca" pod "debf4cec-4ed7-433e-8098-449d7dbc20a6" (UID: "debf4cec-4ed7-433e-8098-449d7dbc20a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:29.692754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.692698 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr" pod "debf4cec-4ed7-433e-8098-449d7dbc20a6" (UID: "debf4cec-4ed7-433e-8098-449d7dbc20a6"). InnerVolumeSpecName "kube-api-access-j5qgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:29.792173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.792086 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:29.792173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.792128 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/debf4cec-4ed7-433e-8098-449d7dbc20a6-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:29.792173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:29.792144 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5qgr\" (UniqueName: \"kubernetes.io/projected/debf4cec-4ed7-433e-8098-449d7dbc20a6-kube-api-access-j5qgr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:30.526880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:30.526851 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" Nov 03 21:21:30.527091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:30.526885 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v" event={"ID":"debf4cec-4ed7-433e-8098-449d7dbc20a6","Type":"ContainerDied","Data":"124187358f2c26fcefca7eeafc7b80ef117dabb7d816bb87a1e02b5e9fdf6b31"} Nov 03 21:21:30.527091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:30.526933 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="124187358f2c26fcefca7eeafc7b80ef117dabb7d816bb87a1e02b5e9fdf6b31" Nov 03 21:21:30.799566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:30.799476 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2"] Nov 03 21:21:37.906111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:37.906076 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb"] Nov 03 21:21:38.407939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.407896 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2"] Nov 03 21:21:38.408440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.408421 5932 cpu_manager.go:414] "containerMap: removing container" podUID="debf4cec-4ed7-433e-8098-449d7dbc20a6" containerName="revoke-keys" Nov 03 21:21:38.408539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.408441 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debf4cec-4ed7-433e-8098-449d7dbc20a6" containerName="revoke-keys" Nov 03 21:21:38.408606 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.408579 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="debf4cec-4ed7-433e-8098-449d7dbc20a6" containerName="revoke-keys" Nov 03 21:21:38.411611 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.411588 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.465662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.465614 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgc2d\" (UniqueName: \"kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.465818 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.465732 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.465818 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.465765 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.468824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.468800 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:21:38.470827 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.470804 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:21:38.471122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.471099 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2"] Nov 03 21:21:38.566972 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.566934 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgc2d\" (UniqueName: \"kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.567141 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.566992 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.567141 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.567017 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.567689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.567651 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.569505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.569481 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.624915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.624882 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgc2d\" (UniqueName: \"kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d\") pod \"maas-api-revoke-sub-f01949aa-qhwb2\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:38.725068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:38.724992 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:39.096844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:39.096762 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2"] Nov 03 21:21:39.103556 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:21:39.103528 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ef152d_2e94_417d_ae48_f9dbfd1088e7.slice/crio-471024289c72baa02c2b5d4df2ba41a84e40ba9cce11064a9c0bad56453472d5 WatchSource:0}: Error finding container 471024289c72baa02c2b5d4df2ba41a84e40ba9cce11064a9c0bad56453472d5: Status 404 returned error can't find the container with id 471024289c72baa02c2b5d4df2ba41a84e40ba9cce11064a9c0bad56453472d5 Nov 03 21:21:39.588689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:39.588651 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" event={"ID":"b2ef152d-2e94-417d-ae48-f9dbfd1088e7","Type":"ContainerStarted","Data":"b4a4944c6cd0f3655cd139860e0cf3823f992c7710ef4651a69d494fdeb9328e"} Nov 03 21:21:39.588873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:39.588749 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" event={"ID":"b2ef152d-2e94-417d-ae48-f9dbfd1088e7","Type":"ContainerStarted","Data":"471024289c72baa02c2b5d4df2ba41a84e40ba9cce11064a9c0bad56453472d5"} Nov 03 21:21:39.698114 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:39.698009 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" podStartSLOduration=1.6979945490000001 podStartE2EDuration="1.697994549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:21:39.696589042 +0000 UTC m=+3107.761830517" watchObservedRunningTime="2026-11-03 21:21:39.697994549 +0000 UTC m=+3107.763236019" Nov 03 21:21:40.596663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:40.596610 5932 generic.go:356] "Generic (PLEG): container finished" podID="b2ef152d-2e94-417d-ae48-f9dbfd1088e7" containerID="b4a4944c6cd0f3655cd139860e0cf3823f992c7710ef4651a69d494fdeb9328e" exitCode=0 Nov 03 21:21:40.596663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:40.596653 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" event={"ID":"b2ef152d-2e94-417d-ae48-f9dbfd1088e7","Type":"ContainerDied","Data":"b4a4944c6cd0f3655cd139860e0cf3823f992c7710ef4651a69d494fdeb9328e"} Nov 03 21:21:41.057697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.057662 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b"] Nov 03 21:21:41.061091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.061066 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e96e7c3d-7cf5b"] Nov 03 21:21:41.734407 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.734381 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:41.798596 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.798556 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca\") pod \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " Nov 03 21:21:41.798794 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.798647 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca\") pod \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " Nov 03 21:21:41.798794 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.798703 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d\" (UniqueName: \"kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d\") pod \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\" (UID: \"b2ef152d-2e94-417d-ae48-f9dbfd1088e7\") " Nov 03 21:21:41.799026 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.799000 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca" pod "b2ef152d-2e94-417d-ae48-f9dbfd1088e7" (UID: "b2ef152d-2e94-417d-ae48-f9dbfd1088e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:21:41.800863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.800843 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca" pod "b2ef152d-2e94-417d-ae48-f9dbfd1088e7" (UID: "b2ef152d-2e94-417d-ae48-f9dbfd1088e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:41.800924 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.800871 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d" pod "b2ef152d-2e94-417d-ae48-f9dbfd1088e7" (UID: "b2ef152d-2e94-417d-ae48-f9dbfd1088e7"). InnerVolumeSpecName "kube-api-access-hgc2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:41.900326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.900291 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:41.900326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.900325 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:41.900533 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:41.900336 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgc2d\" (UniqueName: \"kubernetes.io/projected/b2ef152d-2e94-417d-ae48-f9dbfd1088e7-kube-api-access-hgc2d\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:42.050537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:42.050501 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d6458f-332d-44ef-bed7-c25058966053" path="/var/lib/kubelet/pods/b5d6458f-332d-44ef-bed7-c25058966053/volumes" Nov 03 21:21:42.610107 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:42.610077 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" Nov 03 21:21:42.610271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:42.610113 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2" event={"ID":"b2ef152d-2e94-417d-ae48-f9dbfd1088e7","Type":"ContainerDied","Data":"471024289c72baa02c2b5d4df2ba41a84e40ba9cce11064a9c0bad56453472d5"} Nov 03 21:21:42.610271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:42.610158 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="471024289c72baa02c2b5d4df2ba41a84e40ba9cce11064a9c0bad56453472d5" Nov 03 21:21:42.907281 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:42.907241 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerName="payload-processing" containerID="cri-o://8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf" gracePeriod=25 Nov 03 21:21:44.151322 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:21:44.151286 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:21:45.819994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:45.819926 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" containerID="cri-o://bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e" gracePeriod=45 Nov 03 21:21:46.077688 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.077593 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 21:21:46.139335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139296 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache\") pod \"09e7c8a5-e276-418a-b4d5-474923fbff63\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " Nov 03 21:21:46.139524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139380 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm\") pod \"09e7c8a5-e276-418a-b4d5-474923fbff63\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " Nov 03 21:21:46.139524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139402 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home\") pod \"09e7c8a5-e276-418a-b4d5-474923fbff63\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " Nov 03 21:21:46.139524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139449 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir\") pod \"09e7c8a5-e276-418a-b4d5-474923fbff63\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " Nov 03 21:21:46.139524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139474 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94\" (UniqueName: \"kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94\") pod \"09e7c8a5-e276-418a-b4d5-474923fbff63\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " Nov 03 21:21:46.139524 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139502 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs\") pod \"09e7c8a5-e276-418a-b4d5-474923fbff63\" (UID: \"09e7c8a5-e276-418a-b4d5-474923fbff63\") " Nov 03 21:21:46.139819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139601 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache" pod "09e7c8a5-e276-418a-b4d5-474923fbff63" (UID: "09e7c8a5-e276-418a-b4d5-474923fbff63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:21:46.139819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139701 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir" pod "09e7c8a5-e276-418a-b4d5-474923fbff63" (UID: "09e7c8a5-e276-418a-b4d5-474923fbff63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:21:46.139819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139718 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home" pod "09e7c8a5-e276-418a-b4d5-474923fbff63" (UID: "09e7c8a5-e276-418a-b4d5-474923fbff63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:21:46.139988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139896 5932 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-tmp-dir\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:46.139988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139914 5932 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-model-cache\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:46.139988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.139924 5932 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-home\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:46.141731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.141704 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm" pod "09e7c8a5-e276-418a-b4d5-474923fbff63" (UID: "09e7c8a5-e276-418a-b4d5-474923fbff63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:21:46.141849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.141729 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94" pod "09e7c8a5-e276-418a-b4d5-474923fbff63" (UID: "09e7c8a5-e276-418a-b4d5-474923fbff63"). InnerVolumeSpecName "kube-api-access-78b94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:21:46.141849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.141770 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs" pod "09e7c8a5-e276-418a-b4d5-474923fbff63" (UID: "09e7c8a5-e276-418a-b4d5-474923fbff63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:21:46.241286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.241246 5932 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e7c8a5-e276-418a-b4d5-474923fbff63-dshm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:46.241286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.241279 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78b94\" (UniqueName: \"kubernetes.io/projected/09e7c8a5-e276-418a-b4d5-474923fbff63-kube-api-access-78b94\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:46.241286 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.241290 5932 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e7c8a5-e276-418a-b4d5-474923fbff63-tls-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:21:46.643450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.643408 5932 generic.go:356] "Generic (PLEG): container finished" podID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerID="bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e" exitCode=0 Nov 03 21:21:46.643678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.643456 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" Nov 03 21:21:46.643678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.643501 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" event={"ID":"09e7c8a5-e276-418a-b4d5-474923fbff63","Type":"ContainerDied","Data":"bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e"} Nov 03 21:21:46.643678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.643547 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2" event={"ID":"09e7c8a5-e276-418a-b4d5-474923fbff63","Type":"ContainerDied","Data":"416754cec034ed7665c725a08003e8baac7d766a24fd37b10a596b28aa172636"} Nov 03 21:21:46.643678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.643568 5932 scope.go:122] "RemoveContainer" containerID="bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e" Nov 03 21:21:46.653796 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.653778 5932 scope.go:122] "RemoveContainer" containerID="bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e" Nov 03 21:21:46.654100 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:21:46.654075 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e\": container with ID starting with bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e not found: ID does not exist" containerID="bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e" Nov 03 21:21:46.654169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.654114 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e"} err="failed to get container status \"bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e\": rpc error: code = NotFound desc = could not find container \"bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e\": container with ID starting with bd82a5e26a542a1500ed07085de69c3207b6026ec1082366815ea4874bf2381e not found: ID does not exist" Nov 03 21:21:46.797684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.797645 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2"] Nov 03 21:21:46.804806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:46.804777 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-504eb0/premium-w6-504eb0-kserve-6845dbfd56-hcsc2"] Nov 03 21:21:47.107787 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:47.107687 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:21:48.051377 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:48.051340 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" path="/var/lib/kubelet/pods/09e7c8a5-e276-418a-b4d5-474923fbff63/volumes" Nov 03 21:21:55.635982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:55.635944 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v"] Nov 03 21:21:55.682353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:55.682314 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-504eb0-hqn2v"] Nov 03 21:21:56.050833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:56.050756 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debf4cec-4ed7-433e-8098-449d7dbc20a6" path="/var/lib/kubelet/pods/debf4cec-4ed7-433e-8098-449d7dbc20a6/volumes" Nov 03 21:21:57.108207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:21:57.108167 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:22:00.860977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:00.860935 5932 scope.go:122] "RemoveContainer" containerID="866b75dbb4cc1987de2daa0595af0c26bc31c30bf9bb3b60505c3c6a6c86f2b3" Nov 03 21:22:00.870719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:00.870692 5932 scope.go:122] "RemoveContainer" containerID="23e717196b8d76f05ff41c2dbc3bfdd58c3f91277f9ddb85aa81cfcd56311d2d" Nov 03 21:22:00.884344 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:00.884320 5932 scope.go:122] "RemoveContainer" containerID="11dd445d26d9b3face72c5bc8d50ebb3db6ba80b3268d0dacdfe9fac9ac99709" Nov 03 21:22:03.179146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.179072 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 21:22:03.315727 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.315686 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume\") pod \"108684b1-6fb1-4d9f-863d-7745c1fe6837\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " Nov 03 21:22:03.315933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.315882 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh\" (UniqueName: \"kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh\") pod \"108684b1-6fb1-4d9f-863d-7745c1fe6837\" (UID: \"108684b1-6fb1-4d9f-863d-7745c1fe6837\") " Nov 03 21:22:03.316141 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.316116 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume" pod "108684b1-6fb1-4d9f-863d-7745c1fe6837" (UID: "108684b1-6fb1-4d9f-863d-7745c1fe6837"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:03.318044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.318017 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh" pod "108684b1-6fb1-4d9f-863d-7745c1fe6837" (UID: "108684b1-6fb1-4d9f-863d-7745c1fe6837"). InnerVolumeSpecName "kube-api-access-k28dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:03.417519 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.417483 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/108684b1-6fb1-4d9f-863d-7745c1fe6837-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:03.417519 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.417512 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28dh\" (UniqueName: \"kubernetes.io/projected/108684b1-6fb1-4d9f-863d-7745c1fe6837-kube-api-access-k28dh\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:03.758563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.758528 5932 generic.go:356] "Generic (PLEG): container finished" podID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerID="8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf" exitCode=0 Nov 03 21:22:03.758781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.758600 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" Nov 03 21:22:03.758781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.758637 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" event={"ID":"108684b1-6fb1-4d9f-863d-7745c1fe6837","Type":"ContainerDied","Data":"8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf"} Nov 03 21:22:03.758781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.758709 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb" event={"ID":"108684b1-6fb1-4d9f-863d-7745c1fe6837","Type":"ContainerDied","Data":"a3a729643e982099046a1249a359ede4a967bde60d6893662198c7f4fc9cc37f"} Nov 03 21:22:03.758781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.758738 5932 scope.go:122] "RemoveContainer" containerID="8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf" Nov 03 21:22:03.769186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.769163 5932 scope.go:122] "RemoveContainer" containerID="8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf" Nov 03 21:22:03.769475 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:22:03.769452 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf\": container with ID starting with 8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf not found: ID does not exist" containerID="8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf" Nov 03 21:22:03.769538 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.769485 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf"} err="failed to get container status \"8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf\": rpc error: code = NotFound desc = could not find container \"8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf\": container with ID starting with 8b59786c17094d0c91aa3fa013065ccd930c1fd68ed97584c6d233e7fbe655bf not found: ID does not exist" Nov 03 21:22:03.911498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.911462 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb"] Nov 03 21:22:03.958304 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:03.958267 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-504eb0-7c9d4c9bc-mh6jb"] Nov 03 21:22:04.052508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:04.052429 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" path="/var/lib/kubelet/pods/108684b1-6fb1-4d9f-863d-7745c1fe6837/volumes" Nov 03 21:22:08.398950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.398914 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-6mgs6"] Nov 03 21:22:08.399463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.399129 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" containerID="cri-o://488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131" gracePeriod=30 Nov 03 21:22:08.649116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.649045 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 21:22:08.771332 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.771295 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs\") pod \"dc903517-46d5-4392-83e1-a73e1d159bb0\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " Nov 03 21:22:08.771541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.771407 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk\" (UniqueName: \"kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk\") pod \"dc903517-46d5-4392-83e1-a73e1d159bb0\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " Nov 03 21:22:08.771541 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.771436 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert\") pod \"dc903517-46d5-4392-83e1-a73e1d159bb0\" (UID: \"dc903517-46d5-4392-83e1-a73e1d159bb0\") " Nov 03 21:22:08.774334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.774302 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert" pod "dc903517-46d5-4392-83e1-a73e1d159bb0" (UID: "dc903517-46d5-4392-83e1-a73e1d159bb0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:08.774733 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.774698 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk" pod "dc903517-46d5-4392-83e1-a73e1d159bb0" (UID: "dc903517-46d5-4392-83e1-a73e1d159bb0"). InnerVolumeSpecName "kube-api-access-68vxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:08.774865 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.774786 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs" pod "dc903517-46d5-4392-83e1-a73e1d159bb0" (UID: "dc903517-46d5-4392-83e1-a73e1d159bb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:08.794316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.794278 5932 generic.go:356] "Generic (PLEG): container finished" podID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerID="488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131" exitCode=0 Nov 03 21:22:08.794498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.794325 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" Nov 03 21:22:08.794498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.794365 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" event={"ID":"dc903517-46d5-4392-83e1-a73e1d159bb0","Type":"ContainerDied","Data":"488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131"} Nov 03 21:22:08.794498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.794432 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-6mgs6" event={"ID":"dc903517-46d5-4392-83e1-a73e1d159bb0","Type":"ContainerDied","Data":"65089972c5fb78cc10c20315264b0ae85bea836d29a6bb28b6c65858ca7e0b28"} Nov 03 21:22:08.794498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.794460 5932 scope.go:122] "RemoveContainer" containerID="488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131" Nov 03 21:22:08.807385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.807359 5932 scope.go:122] "RemoveContainer" containerID="03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc" Nov 03 21:22:08.817941 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.817914 5932 scope.go:122] "RemoveContainer" containerID="488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131" Nov 03 21:22:08.818280 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:22:08.818241 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131\": container with ID starting with 488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131 not found: ID does not exist" containerID="488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131" Nov 03 21:22:08.818424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.818286 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131"} err="failed to get container status \"488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131\": rpc error: code = NotFound desc = could not find container \"488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131\": container with ID starting with 488771d04b406bec244175e028d9e9a48269170d163374ce3ee07ea8cffed131 not found: ID does not exist" Nov 03 21:22:08.818424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.818305 5932 scope.go:122] "RemoveContainer" containerID="03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc" Nov 03 21:22:08.818601 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:22:08.818562 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc\": container with ID starting with 03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc not found: ID does not exist" containerID="03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc" Nov 03 21:22:08.818601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.818583 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc"} err="failed to get container status \"03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc\": rpc error: code = NotFound desc = could not find container \"03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc\": container with ID starting with 03e20b2df5cb3e6c4982fa6b42a957d6d472ab29455fbc9ddb1eca3ba13520fc not found: ID does not exist" Nov 03 21:22:08.873235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.873198 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68vxk\" (UniqueName: \"kubernetes.io/projected/dc903517-46d5-4392-83e1-a73e1d159bb0-kube-api-access-68vxk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:08.873235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.873233 5932 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-webhook-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:08.873235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.873243 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc903517-46d5-4392-83e1-a73e1d159bb0-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:08.949809 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.949722 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-6mgs6"] Nov 03 21:22:08.953413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:08.953381 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-6mgs6"] Nov 03 21:22:09.780935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.780897 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-l6zsf"] Nov 03 21:22:09.781335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781302 5932 cpu_manager.go:414] "containerMap: removing container" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" Nov 03 21:22:09.781335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781313 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" Nov 03 21:22:09.781335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781322 5932 cpu_manager.go:414] "containerMap: removing container" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" Nov 03 21:22:09.781335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781327 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781339 5932 cpu_manager.go:414] "containerMap: removing container" podUID="b2ef152d-2e94-417d-ae48-f9dbfd1088e7" containerName="revoke-keys" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781345 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ef152d-2e94-417d-ae48-f9dbfd1088e7" containerName="revoke-keys" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781354 5932 cpu_manager.go:414] "containerMap: removing container" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781359 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781376 5932 cpu_manager.go:414] "containerMap: removing container" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerName="payload-processing" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781381 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerName="payload-processing" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781459 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781466 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e7c8a5-e276-418a-b4d5-474923fbff63" containerName="main" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781472 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ef152d-2e94-417d-ae48-f9dbfd1088e7" containerName="revoke-keys" Nov 03 21:22:09.781475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781478 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" containerName="manager" Nov 03 21:22:09.781806 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.781487 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="108684b1-6fb1-4d9f-863d-7745c1fe6837" containerName="payload-processing" Nov 03 21:22:09.786294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.786271 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.843436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.843397 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-l6zsf"] Nov 03 21:22:09.884858 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.884819 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.885038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.884885 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.885038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.884968 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc89j\" (UniqueName: \"kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.886803 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.886779 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 21:22:09.886925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.886814 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 21:22:09.886925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.886814 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 21:22:09.886925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.886880 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fc9dv\"" Nov 03 21:22:09.985900 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.985865 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.986112 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.985912 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.986112 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.985942 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc89j\" (UniqueName: \"kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.988514 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.988484 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:09.988673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:09.988565 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:10.043702 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:10.043591 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc89j\" (UniqueName: \"kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j\") pod \"maas-controller-7dcb8dd554-l6zsf\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:10.049925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:10.049892 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc903517-46d5-4392-83e1-a73e1d159bb0" path="/var/lib/kubelet/pods/dc903517-46d5-4392-83e1-a73e1d159bb0/volumes" Nov 03 21:22:10.099907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:10.099844 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:10.484302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:10.484263 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-l6zsf"] Nov 03 21:22:10.490336 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:22:10.490300 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff39947_2e89_4957_ac47_8ccd884c9395.slice/crio-40d7afcd74ad813202047b11224d3247c2d9c77af4e687c302765835e9da2f81 WatchSource:0}: Error finding container 40d7afcd74ad813202047b11224d3247c2d9c77af4e687c302765835e9da2f81: Status 404 returned error can't find the container with id 40d7afcd74ad813202047b11224d3247c2d9c77af4e687c302765835e9da2f81 Nov 03 21:22:10.492005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:10.491985 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:22:10.812774 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:10.812672 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" event={"ID":"6ff39947-2e89-4957-ac47-8ccd884c9395","Type":"ContainerStarted","Data":"40d7afcd74ad813202047b11224d3247c2d9c77af4e687c302765835e9da2f81"} Nov 03 21:22:11.827605 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:11.827510 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" event={"ID":"6ff39947-2e89-4957-ac47-8ccd884c9395","Type":"ContainerStarted","Data":"bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9"} Nov 03 21:22:11.828069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:11.827606 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:11.940207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:11.940146 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" podStartSLOduration=1.938824392 podStartE2EDuration="2.94013095s" totalImagesPullingTime="1.001306558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:11.938142942 +0000 UTC m=+3140.003384414" watchObservedRunningTime="2026-11-03 21:22:11.94013095 +0000 UTC m=+3140.005372471" Nov 03 21:22:14.152319 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:22:14.152278 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:22:22.836124 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:22.836090 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:29.099246 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.099207 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6"] Nov 03 21:22:29.103016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.102978 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.144078 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.144039 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747"] Nov 03 21:22:29.147754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.147730 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.160443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160412 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160611 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160454 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160611 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160557 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65pzp\" (UniqueName: \"kubernetes.io/projected/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-kube-api-access-65pzp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160648 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160729 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160831 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160879 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.160981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160962 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.161034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.160997 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.163051 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.163036 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-568z7\"" Nov 03 21:22:29.192770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.192736 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6"] Nov 03 21:22:29.242033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.241997 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747"] Nov 03 21:22:29.262119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262079 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c400677-019d-4644-b45e-92057290a7cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.262335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262136 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.262335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262219 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.262335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262278 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.262335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262321 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.262581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262350 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.262581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262389 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.262581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262417 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdw7d\" (UniqueName: \"kubernetes.io/projected/0c400677-019d-4644-b45e-92057290a7cf-kube-api-access-cdw7d\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.262581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262449 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c400677-019d-4644-b45e-92057290a7cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.262581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262481 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.262581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262543 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262745 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262789 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.263009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262825 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.262929 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263353 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263329 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263435 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263418 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263480 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263465 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263516 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65pzp\" (UniqueName: \"kubernetes.io/projected/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-kube-api-access-65pzp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263573 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c400677-019d-4644-b45e-92057290a7cf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.263691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263617 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.263753 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263729 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.263813 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.263779 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.264928 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.264903 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.265846 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.265820 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.320126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.320091 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.320667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.320647 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65pzp\" (UniqueName: \"kubernetes.io/projected/82930fab-d2aa-49e9-b83e-bcbc34c20ba5-kube-api-access-65pzp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-45kq6\" (UID: \"82930fab-d2aa-49e9-b83e-bcbc34c20ba5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.364489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364403 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364449 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364465 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdw7d\" (UniqueName: \"kubernetes.io/projected/0c400677-019d-4644-b45e-92057290a7cf-kube-api-access-cdw7d\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364484 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c400677-019d-4644-b45e-92057290a7cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364507 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364534 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364586 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c400677-019d-4644-b45e-92057290a7cf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364726 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.364897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364773 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c400677-019d-4644-b45e-92057290a7cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.365131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.364963 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.365131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.365005 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.365131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.365074 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.365244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.365174 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.365443 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.365419 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c400677-019d-4644-b45e-92057290a7cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.366930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.366906 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c400677-019d-4644-b45e-92057290a7cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.367306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.367284 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c400677-019d-4644-b45e-92057290a7cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.416422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.416383 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:29.422130 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.422099 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c400677-019d-4644-b45e-92057290a7cf-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.422874 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.422846 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdw7d\" (UniqueName: \"kubernetes.io/projected/0c400677-019d-4644-b45e-92057290a7cf-kube-api-access-cdw7d\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-lk747\" (UID: \"0c400677-019d-4644-b45e-92057290a7cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.462016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.461968 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:29.844129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.844078 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6"] Nov 03 21:22:29.846929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.846898 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747"] Nov 03 21:22:29.850488 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:22:29.850458 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82930fab_d2aa_49e9_b83e_bcbc34c20ba5.slice/crio-11bda7c0f1a62347f4bac29ab530c9ebdec3aba5cb2944c06d7562cfdb5b44f2 WatchSource:0}: Error finding container 11bda7c0f1a62347f4bac29ab530c9ebdec3aba5cb2944c06d7562cfdb5b44f2: Status 404 returned error can't find the container with id 11bda7c0f1a62347f4bac29ab530c9ebdec3aba5cb2944c06d7562cfdb5b44f2 Nov 03 21:22:29.852665 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.852634 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:22:29.852740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.852699 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:22:29.852740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.852731 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:22:29.889816 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:22:29.889785 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c400677_019d_4644_b45e_92057290a7cf.slice/crio-274a6a23fdca1577e74984ca08f364e0310534eb3e6c68a96d83f391222653ce WatchSource:0}: Error finding container 274a6a23fdca1577e74984ca08f364e0310534eb3e6c68a96d83f391222653ce: Status 404 returned error can't find the container with id 274a6a23fdca1577e74984ca08f364e0310534eb3e6c68a96d83f391222653ce Nov 03 21:22:29.892058 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.892018 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:22:29.892184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.892114 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:22:29.892184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.892163 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:22:29.954937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.954897 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" event={"ID":"0c400677-019d-4644-b45e-92057290a7cf","Type":"ContainerStarted","Data":"274a6a23fdca1577e74984ca08f364e0310534eb3e6c68a96d83f391222653ce"} Nov 03 21:22:29.956952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.956916 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" event={"ID":"82930fab-d2aa-49e9-b83e-bcbc34c20ba5","Type":"ContainerStarted","Data":"8525d28d298897aeaa57242852f2cabf436c9535bb275f2a6b9b1d7e76c326a3"} Nov 03 21:22:29.957132 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:29.956981 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" event={"ID":"82930fab-d2aa-49e9-b83e-bcbc34c20ba5","Type":"ContainerStarted","Data":"11bda7c0f1a62347f4bac29ab530c9ebdec3aba5cb2944c06d7562cfdb5b44f2"} Nov 03 21:22:30.073526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.073460 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podStartSLOduration=1.073442312 podStartE2EDuration="1.073442312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:30.069926612 +0000 UTC m=+3158.135168091" watchObservedRunningTime="2026-11-03 21:22:30.073442312 +0000 UTC m=+3158.138683784" Nov 03 21:22:30.417880 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.417844 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:30.430393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.430355 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:30.430772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.430741 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:30.851745 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.840555 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq"] Nov 03 21:22:30.859648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.857027 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.884650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.881153 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq"] Nov 03 21:22:30.887004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.886974 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.887204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.887183 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld42s\" (UniqueName: \"kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.887380 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.887364 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.972647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.971923 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:22:30.972647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.972114 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:22:30.982660 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.982014 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" event={"ID":"0c400677-019d-4644-b45e-92057290a7cf","Type":"ContainerStarted","Data":"41a4b0c524421d2ee773e9f67ab6e4eb43a469cb271bd514b270f0ebe2c46ec5"} Nov 03 21:22:30.987987 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.987954 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.988172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.988041 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.988172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.988061 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld42s\" (UniqueName: \"kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.988828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.988798 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:30.991272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:30.991246 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:31.074647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.073788 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld42s\" (UniqueName: \"kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s\") pod \"maas-api-revoke-sub-85237e45-64cxq\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:31.101593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.101535 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podStartSLOduration=2.101520767 podStartE2EDuration="2.101520767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:31.098875295 +0000 UTC m=+3159.164116767" watchObservedRunningTime="2026-11-03 21:22:31.101520767 +0000 UTC m=+3159.166762239" Nov 03 21:22:31.194123 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.194088 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:31.416829 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.416788 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:31.417004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.416875 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:31.472707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.471679 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:31.475653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.474755 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:31.475653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.474812 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:31.642648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.632843 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq"] Nov 03 21:22:31.989554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:31.989513 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" event={"ID":"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e","Type":"ContainerStarted","Data":"f76ae9460dbab30dede06ff6eb31f0c1e1d7dc3896cedf34f9265559dcd71658"} Nov 03 21:22:32.417258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:32.417210 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:32.417456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:32.417278 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:32.474649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:32.473887 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:32.474649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:32.473961 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:32.996689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:32.996648 5932 generic.go:356] "Generic (PLEG): container finished" podID="e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" containerID="dfec9e15f6a6f534fbcb723c45c1bca788664ebc010103ff5c6f27cd4130a5e1" exitCode=0 Nov 03 21:22:32.996906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:32.996723 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" event={"ID":"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e","Type":"ContainerDied","Data":"dfec9e15f6a6f534fbcb723c45c1bca788664ebc010103ff5c6f27cd4130a5e1"} Nov 03 21:22:33.067638 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.067577 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv"] Nov 03 21:22:33.067925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.067907 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg"] Nov 03 21:22:33.093639 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.093592 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0137c46e-49mwv"] Nov 03 21:22:33.096222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.096186 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8985ac3-kwdgg"] Nov 03 21:22:33.417492 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.417441 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:33.417800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.417511 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:33.467652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.467075 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:33.467652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:33.467139 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:34.077636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.077594 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2aae20-9e84-42a3-af2a-67a72c54068e" path="/var/lib/kubelet/pods/2d2aae20-9e84-42a3-af2a-67a72c54068e/volumes" Nov 03 21:22:34.078133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.078074 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4408b4-fe22-4c79-9b42-f12eac2292ac" path="/var/lib/kubelet/pods/de4408b4-fe22-4c79-9b42-f12eac2292ac/volumes" Nov 03 21:22:34.277361 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.277286 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:34.319522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.319484 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca\") pod \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " Nov 03 21:22:34.319762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.319607 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s\" (UniqueName: \"kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s\") pod \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " Nov 03 21:22:34.319762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.319704 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca\") pod \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\" (UID: \"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e\") " Nov 03 21:22:34.320247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.320223 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca" pod "e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" (UID: "e547156d-b6e6-4b8e-9e34-2faeeba8ea1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:22:34.349978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.346769 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca" pod "e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" (UID: "e547156d-b6e6-4b8e-9e34-2faeeba8ea1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:34.371928 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.360331 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s" pod "e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" (UID: "e547156d-b6e6-4b8e-9e34-2faeeba8ea1e"). InnerVolumeSpecName "kube-api-access-ld42s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:34.416921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.416881 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:34.417097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.416949 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:34.435649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.428382 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:34.435649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.428425 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld42s\" (UniqueName: \"kubernetes.io/projected/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-kube-api-access-ld42s\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:34.435649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.428442 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:34.476176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.476061 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:34.476176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:34.476120 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:35.011109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.011078 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" Nov 03 21:22:35.011294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.011117 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq" event={"ID":"e547156d-b6e6-4b8e-9e34-2faeeba8ea1e","Type":"ContainerDied","Data":"f76ae9460dbab30dede06ff6eb31f0c1e1d7dc3896cedf34f9265559dcd71658"} Nov 03 21:22:35.011294 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.011193 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f76ae9460dbab30dede06ff6eb31f0c1e1d7dc3896cedf34f9265559dcd71658" Nov 03 21:22:35.416911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.416876 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:35.417326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.416931 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:35.467800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.467760 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:35.468104 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:35.468072 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:36.417027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:36.416985 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:36.417558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:36.417056 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:36.463350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:36.463313 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:36.463590 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:36.463370 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:37.417650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:37.417593 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" start-of-body= Nov 03 21:22:37.418074 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:37.417682 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": dial tcp 10.132.1.32:15021: connect: connection refused" Nov 03 21:22:37.463594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:37.463552 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 21:22:37.463814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:37.463644 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 21:22:39.417692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:39.417646 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:22:39.418251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:39.417711 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:22:39.504847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:39.504808 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:22:39.505055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:39.504885 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:22:40.460584 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.460507 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:22:40.461318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.461283 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:22:40.546934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.546879 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:22:40.547333 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.547290 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:22:40.687581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.687541 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:40.688014 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.687975 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:40.940714 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:40.940666 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" Nov 03 21:22:41.151495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:41.151455 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:41.151988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:41.151958 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:41.159857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:41.159831 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" Nov 03 21:22:43.438854 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.438819 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-l6zsf"] Nov 03 21:22:43.439335 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.439021 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" podUID="6ff39947-2e89-4957-ac47-8ccd884c9395" containerName="manager" containerID="cri-o://bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9" gracePeriod=30 Nov 03 21:22:43.701271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.701205 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:43.866432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.866401 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j\" (UniqueName: \"kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j\") pod \"6ff39947-2e89-4957-ac47-8ccd884c9395\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " Nov 03 21:22:43.866643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.866442 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs\") pod \"6ff39947-2e89-4957-ac47-8ccd884c9395\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " Nov 03 21:22:43.866643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.866482 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert\") pod \"6ff39947-2e89-4957-ac47-8ccd884c9395\" (UID: \"6ff39947-2e89-4957-ac47-8ccd884c9395\") " Nov 03 21:22:43.868737 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.868695 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert" pod "6ff39947-2e89-4957-ac47-8ccd884c9395" (UID: "6ff39947-2e89-4957-ac47-8ccd884c9395"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:43.868892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.868783 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs" pod "6ff39947-2e89-4957-ac47-8ccd884c9395" (UID: "6ff39947-2e89-4957-ac47-8ccd884c9395"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:22:43.868892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.868875 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j" pod "6ff39947-2e89-4957-ac47-8ccd884c9395" (UID: "6ff39947-2e89-4957-ac47-8ccd884c9395"). InnerVolumeSpecName "kube-api-access-sc89j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:22:43.968217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.968123 5932 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-webhook-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:43.968217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.968160 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc89j\" (UniqueName: \"kubernetes.io/projected/6ff39947-2e89-4957-ac47-8ccd884c9395-kube-api-access-sc89j\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:43.968217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:43.968170 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ff39947-2e89-4957-ac47-8ccd884c9395-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:22:44.088611 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.088575 5932 generic.go:356] "Generic (PLEG): container finished" podID="6ff39947-2e89-4957-ac47-8ccd884c9395" containerID="bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9" exitCode=0 Nov 03 21:22:44.088815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.088635 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" Nov 03 21:22:44.088815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.088670 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" event={"ID":"6ff39947-2e89-4957-ac47-8ccd884c9395","Type":"ContainerDied","Data":"bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9"} Nov 03 21:22:44.088815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.088731 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-l6zsf" event={"ID":"6ff39947-2e89-4957-ac47-8ccd884c9395","Type":"ContainerDied","Data":"40d7afcd74ad813202047b11224d3247c2d9c77af4e687c302765835e9da2f81"} Nov 03 21:22:44.088815 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.088760 5932 scope.go:122] "RemoveContainer" containerID="bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9" Nov 03 21:22:44.098446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.098414 5932 scope.go:122] "RemoveContainer" containerID="bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9" Nov 03 21:22:44.098750 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:22:44.098727 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9\": container with ID starting with bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9 not found: ID does not exist" containerID="bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9" Nov 03 21:22:44.098832 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.098757 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9"} err="failed to get container status \"bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9\": rpc error: code = NotFound desc = could not find container \"bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9\": container with ID starting with bd51452a5275724f415b3e909233a9d642d5391a8210c611396b500a75bbcdb9 not found: ID does not exist" Nov 03 21:22:44.151377 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:22:44.151337 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:22:44.247321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.247224 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-l6zsf"] Nov 03 21:22:44.293424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:44.293388 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-l6zsf"] Nov 03 21:22:45.679687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.679651 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-dczcz"] Nov 03 21:22:45.680097 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.680083 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" containerName="revoke-keys" Nov 03 21:22:45.680142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.680097 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" containerName="revoke-keys" Nov 03 21:22:45.680142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.680113 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6ff39947-2e89-4957-ac47-8ccd884c9395" containerName="manager" Nov 03 21:22:45.680142 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.680119 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff39947-2e89-4957-ac47-8ccd884c9395" containerName="manager" Nov 03 21:22:45.680253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.680185 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" containerName="revoke-keys" Nov 03 21:22:45.680253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.680194 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff39947-2e89-4957-ac47-8ccd884c9395" containerName="manager" Nov 03 21:22:45.690224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.690191 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.740085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.740051 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-dczcz"] Nov 03 21:22:45.783475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.783437 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 21:22:45.785123 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.785099 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-fc9dv\"" Nov 03 21:22:45.785183 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.785099 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 21:22:45.785183 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.785139 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 21:22:45.887155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.887115 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4166cdea-dbd8-4a99-b275-ece179b45b9a-webhook-cert\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.887155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.887156 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4166cdea-dbd8-4a99-b275-ece179b45b9a-metrics-certs\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.887370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.887253 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd946\" (UniqueName: \"kubernetes.io/projected/4166cdea-dbd8-4a99-b275-ece179b45b9a-kube-api-access-pd946\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.987967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.987874 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4166cdea-dbd8-4a99-b275-ece179b45b9a-metrics-certs\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.987967 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.987948 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd946\" (UniqueName: \"kubernetes.io/projected/4166cdea-dbd8-4a99-b275-ece179b45b9a-kube-api-access-pd946\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.988236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.988068 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4166cdea-dbd8-4a99-b275-ece179b45b9a-webhook-cert\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.990419 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.990392 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4166cdea-dbd8-4a99-b275-ece179b45b9a-webhook-cert\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:45.990528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:45.990416 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4166cdea-dbd8-4a99-b275-ece179b45b9a-metrics-certs\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:46.046023 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:46.045980 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd946\" (UniqueName: \"kubernetes.io/projected/4166cdea-dbd8-4a99-b275-ece179b45b9a-kube-api-access-pd946\") pod \"maas-controller-7dcb8dd554-dczcz\" (UID: \"4166cdea-dbd8-4a99-b275-ece179b45b9a\") " pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:46.050455 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:46.050425 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff39947-2e89-4957-ac47-8ccd884c9395" path="/var/lib/kubelet/pods/6ff39947-2e89-4957-ac47-8ccd884c9395/volumes" Nov 03 21:22:46.303886 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:46.303792 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:46.678704 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:46.678668 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7dcb8dd554-dczcz"] Nov 03 21:22:46.713353 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:22:46.713322 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4166cdea_dbd8_4a99_b275_ece179b45b9a.slice/crio-0d398c0ea9b386891ccc765b3e23907ee7c8a516006eb07d6fbb8875b5ea64e8 WatchSource:0}: Error finding container 0d398c0ea9b386891ccc765b3e23907ee7c8a516006eb07d6fbb8875b5ea64e8: Status 404 returned error can't find the container with id 0d398c0ea9b386891ccc765b3e23907ee7c8a516006eb07d6fbb8875b5ea64e8 Nov 03 21:22:47.109671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:47.109575 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" event={"ID":"4166cdea-dbd8-4a99-b275-ece179b45b9a","Type":"ContainerStarted","Data":"0d398c0ea9b386891ccc765b3e23907ee7c8a516006eb07d6fbb8875b5ea64e8"} Nov 03 21:22:48.117356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:48.117256 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" event={"ID":"4166cdea-dbd8-4a99-b275-ece179b45b9a","Type":"ContainerStarted","Data":"88e12a47068110ffcacc76527a13bfeeaf251112f833bc58aa1468216959ae69"} Nov 03 21:22:48.117356 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:48.117317 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:22:48.229687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:48.229615 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" podStartSLOduration=2.30666334 podStartE2EDuration="3.22960071s" totalImagesPullingTime="922.93737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:22:48.228997102 +0000 UTC m=+3176.294238574" watchObservedRunningTime="2026-11-03 21:22:48.22960071 +0000 UTC m=+3176.294842184" Nov 03 21:22:59.126292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:22:59.126257 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7dcb8dd554-dczcz" Nov 03 21:23:00.925417 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:00.925380 5932 scope.go:122] "RemoveContainer" containerID="937823ec70564ecfa573d8ee5ecb7f6e3106a15fd53886f141167704a288fbb9" Nov 03 21:23:00.935194 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:00.935170 5932 scope.go:122] "RemoveContainer" containerID="fdab67556533f666e2840c21b33af44505c890b2475b4a9556db4a64342d547b" Nov 03 21:23:00.943740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:00.943711 5932 scope.go:122] "RemoveContainer" containerID="8da7e1fb8282501fb59171688a7a333b83783799d60a7ad76c1149219270eac2" Nov 03 21:23:14.151272 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:23:14.151232 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:23:41.064210 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.064169 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v"] Nov 03 21:23:41.068412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.068385 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.122996 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.122954 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v"] Nov 03 21:23:41.156135 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.156091 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.156329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.156152 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcp59\" (UniqueName: \"kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.156329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.156238 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.165330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.165298 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:23:41.165436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.165336 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:23:41.257575 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.257535 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.257820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.257667 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.257820 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.257762 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcp59\" (UniqueName: \"kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.258361 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.258332 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.260372 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.260349 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.315762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.315681 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcp59\" (UniqueName: \"kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59\") pod \"maas-api-revoke-sub-c1e271a9-8ll7v\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.382372 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.382333 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:41.756280 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:41.756248 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v"] Nov 03 21:23:41.762592 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:23:41.762563 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3589b691_f94d_4d56_af28_1423b5993834.slice/crio-e689fbc0fc10db62a802e294bb5e645895f40c85b6dc8613f8bcb29afb3baa81 WatchSource:0}: Error finding container e689fbc0fc10db62a802e294bb5e645895f40c85b6dc8613f8bcb29afb3baa81: Status 404 returned error can't find the container with id e689fbc0fc10db62a802e294bb5e645895f40c85b6dc8613f8bcb29afb3baa81 Nov 03 21:23:42.441125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:42.441090 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" event={"ID":"3589b691-f94d-4d56-af28-1423b5993834","Type":"ContainerStarted","Data":"0ff419c12a77bf18738991896fd110c6a91be04e77ff879c5992fbc6e5db7178"} Nov 03 21:23:42.441586 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:42.441147 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" event={"ID":"3589b691-f94d-4d56-af28-1423b5993834","Type":"ContainerStarted","Data":"e689fbc0fc10db62a802e294bb5e645895f40c85b6dc8613f8bcb29afb3baa81"} Nov 03 21:23:42.550570 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:42.550510 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" podStartSLOduration=1.55049449 podStartE2EDuration="1.55049449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:42.548564654 +0000 UTC m=+3230.613806127" watchObservedRunningTime="2026-11-03 21:23:42.55049449 +0000 UTC m=+3230.615735962" Nov 03 21:23:43.450013 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:43.449976 5932 generic.go:356] "Generic (PLEG): container finished" podID="3589b691-f94d-4d56-af28-1423b5993834" containerID="0ff419c12a77bf18738991896fd110c6a91be04e77ff879c5992fbc6e5db7178" exitCode=0 Nov 03 21:23:43.450463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:43.450064 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" event={"ID":"3589b691-f94d-4d56-af28-1423b5993834","Type":"ContainerDied","Data":"0ff419c12a77bf18738991896fd110c6a91be04e77ff879c5992fbc6e5db7178"} Nov 03 21:23:44.150328 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:23:44.150284 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:23:44.597648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.597597 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:44.692005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.691967 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca\") pod \"3589b691-f94d-4d56-af28-1423b5993834\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " Nov 03 21:23:44.692005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.692009 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59\" (UniqueName: \"kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59\") pod \"3589b691-f94d-4d56-af28-1423b5993834\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " Nov 03 21:23:44.692236 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.692067 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca\") pod \"3589b691-f94d-4d56-af28-1423b5993834\" (UID: \"3589b691-f94d-4d56-af28-1423b5993834\") " Nov 03 21:23:44.692517 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.692489 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca" pod "3589b691-f94d-4d56-af28-1423b5993834" (UID: "3589b691-f94d-4d56-af28-1423b5993834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:23:44.694216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.694188 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca" pod "3589b691-f94d-4d56-af28-1423b5993834" (UID: "3589b691-f94d-4d56-af28-1423b5993834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:23:44.694329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.694191 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59" pod "3589b691-f94d-4d56-af28-1423b5993834" (UID: "3589b691-f94d-4d56-af28-1423b5993834"). InnerVolumeSpecName "kube-api-access-vcp59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:23:44.698424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.698392 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw"] Nov 03 21:23:44.699179 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.699158 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3589b691-f94d-4d56-af28-1423b5993834" containerName="revoke-keys" Nov 03 21:23:44.699231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.699184 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3589b691-f94d-4d56-af28-1423b5993834" containerName="revoke-keys" Nov 03 21:23:44.699338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.699326 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3589b691-f94d-4d56-af28-1423b5993834" containerName="revoke-keys" Nov 03 21:23:44.703125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.703108 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.780401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.780362 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw"] Nov 03 21:23:44.793456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793419 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37560725-01f3-4f64-a350-31463f2bb770-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793461 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793545 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwsw\" (UniqueName: \"kubernetes.io/projected/37560725-01f3-4f64-a350-31463f2bb770-kube-api-access-qmwsw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793581 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793641 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793597 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37560725-01f3-4f64-a350-31463f2bb770-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793722 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37560725-01f3-4f64-a350-31463f2bb770-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793779 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793875 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793816 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.793945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.793930 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.794042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.794030 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3589b691-f94d-4d56-af28-1423b5993834-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:44.794079 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.794045 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcp59\" (UniqueName: \"kubernetes.io/projected/3589b691-f94d-4d56-af28-1423b5993834-kube-api-access-vcp59\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:44.794079 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.794058 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3589b691-f94d-4d56-af28-1423b5993834-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:23:44.895274 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895206 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37560725-01f3-4f64-a350-31463f2bb770-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.895673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895326 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.895673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895349 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.895673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895429 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.895673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895545 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37560725-01f3-4f64-a350-31463f2bb770-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.895673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895578 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.895673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895616 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwsw\" (UniqueName: \"kubernetes.io/projected/37560725-01f3-4f64-a350-31463f2bb770-kube-api-access-qmwsw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.896125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895675 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.896125 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.895703 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37560725-01f3-4f64-a350-31463f2bb770-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.896526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.896499 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.896731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.896555 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.896859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.896728 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.896859 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.896783 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.897359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.897337 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37560725-01f3-4f64-a350-31463f2bb770-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.899027 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.898976 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37560725-01f3-4f64-a350-31463f2bb770-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.899137 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.899088 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37560725-01f3-4f64-a350-31463f2bb770-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.953155 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.953125 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37560725-01f3-4f64-a350-31463f2bb770-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:44.953316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:44.953257 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwsw\" (UniqueName: \"kubernetes.io/projected/37560725-01f3-4f64-a350-31463f2bb770-kube-api-access-qmwsw\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw\" (UID: \"37560725-01f3-4f64-a350-31463f2bb770\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:45.018691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.018586 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:45.445287 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.445244 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw"] Nov 03 21:23:45.452743 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:23:45.452706 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37560725_01f3_4f64_a350_31463f2bb770.slice/crio-cc50605249d0b47dbbc19085af2233bb9ed526385a5a2cd0d6fe004b47bf9f4a WatchSource:0}: Error finding container cc50605249d0b47dbbc19085af2233bb9ed526385a5a2cd0d6fe004b47bf9f4a: Status 404 returned error can't find the container with id cc50605249d0b47dbbc19085af2233bb9ed526385a5a2cd0d6fe004b47bf9f4a Nov 03 21:23:45.455031 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.454993 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:23:45.455126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.455068 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:23:45.455126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.455096 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:23:45.465457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.465417 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" event={"ID":"37560725-01f3-4f64-a350-31463f2bb770","Type":"ContainerStarted","Data":"cc50605249d0b47dbbc19085af2233bb9ed526385a5a2cd0d6fe004b47bf9f4a"} Nov 03 21:23:45.466930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.466907 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" Nov 03 21:23:45.467050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.466935 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v" event={"ID":"3589b691-f94d-4d56-af28-1423b5993834","Type":"ContainerDied","Data":"e689fbc0fc10db62a802e294bb5e645895f40c85b6dc8613f8bcb29afb3baa81"} Nov 03 21:23:45.467050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:45.466985 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e689fbc0fc10db62a802e294bb5e645895f40c85b6dc8613f8bcb29afb3baa81" Nov 03 21:23:46.474416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:46.474379 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" event={"ID":"37560725-01f3-4f64-a350-31463f2bb770","Type":"ContainerStarted","Data":"7713e13067f6a4470d4ad388ce80b2bafb924c4973bb8817ceee35738f3dfe12"} Nov 03 21:23:46.604709 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:46.604650 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podStartSLOduration=2.604609614 podStartE2EDuration="2.604609614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:23:46.604350915 +0000 UTC m=+3234.669592388" watchObservedRunningTime="2026-11-03 21:23:46.604609614 +0000 UTC m=+3234.669851086" Nov 03 21:23:47.019168 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:47.019132 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:47.043906 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:47.043874 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 03 21:23:47.044230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:47.044185 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podUID="37560725-01f3-4f64-a350-31463f2bb770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 03 21:23:48.019915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:48.019880 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 03 21:23:48.020315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:48.019940 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podUID="37560725-01f3-4f64-a350-31463f2bb770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 03 21:23:49.018956 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:49.018923 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 03 21:23:49.019146 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:49.018976 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podUID="37560725-01f3-4f64-a350-31463f2bb770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 03 21:23:50.019470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:50.019432 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 03 21:23:50.019908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:50.019492 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podUID="37560725-01f3-4f64-a350-31463f2bb770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 03 21:23:51.019528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:51.019493 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 03 21:23:51.019963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:51.019549 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podUID="37560725-01f3-4f64-a350-31463f2bb770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 03 21:23:52.019923 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:52.019879 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" start-of-body= Nov 03 21:23:52.020354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:52.019949 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" podUID="37560725-01f3-4f64-a350-31463f2bb770" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.37:15021/healthz/ready\": dial tcp 10.132.1.37:15021: connect: connection refused" Nov 03 21:23:53.785776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:53.785732 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:53.786247 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:53.786172 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:23:53.787156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:23:53.787132 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw" Nov 03 21:24:00.992648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:00.991843 5932 scope.go:122] "RemoveContainer" containerID="45c279828300bf5c4842f107539455bb4f6e1d622d7f2e27e95104390ce50872" Nov 03 21:24:01.004658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:01.004608 5932 scope.go:122] "RemoveContainer" containerID="fb80879a872fceaabcb2b533042bd8157f95921d8a714bf71775051572c8461b" Nov 03 21:24:01.013771 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:01.013748 5932 scope.go:122] "RemoveContainer" containerID="4c79d907b4ba7bc63811c3a9b5623afed8aebe55bc8ec297356d950145b2f92b" Nov 03 21:24:12.656505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.656418 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq"] Nov 03 21:24:12.662098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.662068 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.721302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.721262 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.721486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.721330 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.721486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.721440 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45hr\" (UniqueName: \"kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.756984 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.756942 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:12.757199 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.757032 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:24:12.759663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.759617 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq"] Nov 03 21:24:12.822458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.822422 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.822690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.822470 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.822690 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.822491 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45hr\" (UniqueName: \"kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.823202 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.823179 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.824920 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.824896 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.879867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.879835 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45hr\" (UniqueName: \"kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr\") pod \"maas-api-revoke-sub-2e1f5b5f-fvvsq\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:12.976298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:12.976211 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:13.346320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:13.346235 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq"] Nov 03 21:24:13.352708 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:24:13.352679 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd0e2a8_430d_4308_9581_d91ce348706b.slice/crio-4a4172066eb63d8ed482a241c7e52ab845c0730af62444317eb7e215ac6723ee WatchSource:0}: Error finding container 4a4172066eb63d8ed482a241c7e52ab845c0730af62444317eb7e215ac6723ee: Status 404 returned error can't find the container with id 4a4172066eb63d8ed482a241c7e52ab845c0730af62444317eb7e215ac6723ee Nov 03 21:24:13.672357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:13.672315 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" event={"ID":"ccd0e2a8-430d-4308-9581-d91ce348706b","Type":"ContainerStarted","Data":"3d59f6c51a169226237ffcadd468f4d84b4e772519a42f279201ee46dd543d96"} Nov 03 21:24:13.672834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:13.672379 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" event={"ID":"ccd0e2a8-430d-4308-9581-d91ce348706b","Type":"ContainerStarted","Data":"4a4172066eb63d8ed482a241c7e52ab845c0730af62444317eb7e215ac6723ee"} Nov 03 21:24:13.782350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:13.782296 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" podStartSLOduration=1.782279878 podStartE2EDuration="1.782279878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:24:13.78009364 +0000 UTC m=+3261.845335112" watchObservedRunningTime="2026-11-03 21:24:13.782279878 +0000 UTC m=+3261.847521350" Nov 03 21:24:14.150398 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:24:14.150359 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:24:14.680050 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:14.680010 5932 generic.go:356] "Generic (PLEG): container finished" podID="ccd0e2a8-430d-4308-9581-d91ce348706b" containerID="3d59f6c51a169226237ffcadd468f4d84b4e772519a42f279201ee46dd543d96" exitCode=0 Nov 03 21:24:14.680477 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:14.680077 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" event={"ID":"ccd0e2a8-430d-4308-9581-d91ce348706b","Type":"ContainerDied","Data":"3d59f6c51a169226237ffcadd468f4d84b4e772519a42f279201ee46dd543d96"} Nov 03 21:24:15.821359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.821335 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:15.956494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.956415 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca\") pod \"ccd0e2a8-430d-4308-9581-d91ce348706b\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " Nov 03 21:24:15.956494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.956477 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca\") pod \"ccd0e2a8-430d-4308-9581-d91ce348706b\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " Nov 03 21:24:15.956811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.956641 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr\" (UniqueName: \"kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr\") pod \"ccd0e2a8-430d-4308-9581-d91ce348706b\" (UID: \"ccd0e2a8-430d-4308-9581-d91ce348706b\") " Nov 03 21:24:15.956891 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.956862 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca" pod "ccd0e2a8-430d-4308-9581-d91ce348706b" (UID: "ccd0e2a8-430d-4308-9581-d91ce348706b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:15.956976 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.956960 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:15.958938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.958894 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca" pod "ccd0e2a8-430d-4308-9581-d91ce348706b" (UID: "ccd0e2a8-430d-4308-9581-d91ce348706b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:15.959086 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:15.959054 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr" pod "ccd0e2a8-430d-4308-9581-d91ce348706b" (UID: "ccd0e2a8-430d-4308-9581-d91ce348706b"). InnerVolumeSpecName "kube-api-access-v45hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:16.057761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:16.057731 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45hr\" (UniqueName: \"kubernetes.io/projected/ccd0e2a8-430d-4308-9581-d91ce348706b-kube-api-access-v45hr\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:16.057761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:16.057758 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccd0e2a8-430d-4308-9581-d91ce348706b-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:16.693061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:16.693032 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" Nov 03 21:24:16.693061 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:16.693041 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq" event={"ID":"ccd0e2a8-430d-4308-9581-d91ce348706b","Type":"ContainerDied","Data":"4a4172066eb63d8ed482a241c7e52ab845c0730af62444317eb7e215ac6723ee"} Nov 03 21:24:16.693291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:16.693086 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a4172066eb63d8ed482a241c7e52ab845c0730af62444317eb7e215ac6723ee" Nov 03 21:24:25.120808 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.120769 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz"] Nov 03 21:24:25.121380 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.121239 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ccd0e2a8-430d-4308-9581-d91ce348706b" containerName="revoke-keys" Nov 03 21:24:25.121380 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.121250 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd0e2a8-430d-4308-9581-d91ce348706b" containerName="revoke-keys" Nov 03 21:24:25.121380 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.121322 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd0e2a8-430d-4308-9581-d91ce348706b" containerName="revoke-keys" Nov 03 21:24:25.124434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.124412 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.175656 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.175584 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:24:25.221686 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.221647 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:24:25.224065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.224042 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz"] Nov 03 21:24:25.257200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.257163 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.257392 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.257217 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.257392 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.257370 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sktc8\" (UniqueName: \"kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.358288 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.358251 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sktc8\" (UniqueName: \"kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.358501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.358335 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.358501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.358360 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.359100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.359075 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.360992 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.360970 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.417350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.417320 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sktc8\" (UniqueName: \"kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8\") pod \"maas-api-revoke-sub-ec843018-q94gz\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.438208 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.438174 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:25.810817 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:25.810729 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz"] Nov 03 21:24:25.818673 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:24:25.817016 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2685431_119c_4a64_b40c_99dae2f04a6c.slice/crio-8a82a1c8ec203124d66a592569e3390a5d4c33df0e00ccd8282350125432dcf8 WatchSource:0}: Error finding container 8a82a1c8ec203124d66a592569e3390a5d4c33df0e00ccd8282350125432dcf8: Status 404 returned error can't find the container with id 8a82a1c8ec203124d66a592569e3390a5d4c33df0e00ccd8282350125432dcf8 Nov 03 21:24:26.753765 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:26.753724 5932 generic.go:356] "Generic (PLEG): container finished" podID="e2685431-119c-4a64-b40c-99dae2f04a6c" containerID="f6010881cc6baf76384b3002b5ee199bbf2a5ffa0d8abd9dc848f151f106fc56" exitCode=0 Nov 03 21:24:26.754276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:26.753805 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" event={"ID":"e2685431-119c-4a64-b40c-99dae2f04a6c","Type":"ContainerDied","Data":"f6010881cc6baf76384b3002b5ee199bbf2a5ffa0d8abd9dc848f151f106fc56"} Nov 03 21:24:26.754276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:26.753870 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" event={"ID":"e2685431-119c-4a64-b40c-99dae2f04a6c","Type":"ContainerStarted","Data":"8a82a1c8ec203124d66a592569e3390a5d4c33df0e00ccd8282350125432dcf8"} Nov 03 21:24:27.901717 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:27.901687 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:28.085973 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.085876 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca\") pod \"e2685431-119c-4a64-b40c-99dae2f04a6c\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " Nov 03 21:24:28.086157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.085983 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca\") pod \"e2685431-119c-4a64-b40c-99dae2f04a6c\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " Nov 03 21:24:28.086157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.086069 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8\" (UniqueName: \"kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8\") pod \"e2685431-119c-4a64-b40c-99dae2f04a6c\" (UID: \"e2685431-119c-4a64-b40c-99dae2f04a6c\") " Nov 03 21:24:28.086359 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.086334 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca" pod "e2685431-119c-4a64-b40c-99dae2f04a6c" (UID: "e2685431-119c-4a64-b40c-99dae2f04a6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:24:28.086467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.086448 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:28.088337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.088306 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca" pod "e2685431-119c-4a64-b40c-99dae2f04a6c" (UID: "e2685431-119c-4a64-b40c-99dae2f04a6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:24:28.088337 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.088312 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8" pod "e2685431-119c-4a64-b40c-99dae2f04a6c" (UID: "e2685431-119c-4a64-b40c-99dae2f04a6c"). InnerVolumeSpecName "kube-api-access-sktc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:24:28.187921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.187878 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2685431-119c-4a64-b40c-99dae2f04a6c-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:28.187921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.187915 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sktc8\" (UniqueName: \"kubernetes.io/projected/e2685431-119c-4a64-b40c-99dae2f04a6c-kube-api-access-sktc8\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:24:28.767465 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.767430 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" Nov 03 21:24:28.767465 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.767454 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz" event={"ID":"e2685431-119c-4a64-b40c-99dae2f04a6c","Type":"ContainerDied","Data":"8a82a1c8ec203124d66a592569e3390a5d4c33df0e00ccd8282350125432dcf8"} Nov 03 21:24:28.767715 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:28.767508 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a82a1c8ec203124d66a592569e3390a5d4c33df0e00ccd8282350125432dcf8" Nov 03 21:24:44.152200 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:24:44.152160 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:24:46.102160 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:46.101796 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r"] Nov 03 21:24:46.102160 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:46.101848 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3005de46-qzw7r"] Nov 03 21:24:48.059810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:48.059772 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9b483b-3187-4575-b56d-48d2e929d4dd" path="/var/lib/kubelet/pods/0f9b483b-3187-4575-b56d-48d2e929d4dd/volumes" Nov 03 21:24:56.240559 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:56.240529 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:24:56.256548 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:24:56.256518 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:25:01.046384 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:01.046349 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt"] Nov 03 21:25:01.092456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:01.092417 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fac3c8c-bbwdt"] Nov 03 21:25:02.051172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:02.051134 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b26eeef-96df-4849-aeb4-654617e15fb0" path="/var/lib/kubelet/pods/2b26eeef-96df-4849-aeb4-654617e15fb0/volumes" Nov 03 21:25:14.150946 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:25:14.150903 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:25:23.732649 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.732598 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4"] Nov 03 21:25:23.733182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.733075 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e2685431-119c-4a64-b40c-99dae2f04a6c" containerName="revoke-keys" Nov 03 21:25:23.733182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.733087 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2685431-119c-4a64-b40c-99dae2f04a6c" containerName="revoke-keys" Nov 03 21:25:23.733182 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.733165 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2685431-119c-4a64-b40c-99dae2f04a6c" containerName="revoke-keys" Nov 03 21:25:23.737687 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.737666 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.769471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.769433 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxwdn\" (UniqueName: \"kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.769682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.769545 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.769682 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.769585 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.833752 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.833706 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:25:23.833968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.833762 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:25:23.836173 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.836148 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4"] Nov 03 21:25:23.870258 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.870223 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.870457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.870265 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.870457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.870329 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxwdn\" (UniqueName: \"kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.871071 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.871044 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.872769 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.872745 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:23.928444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:23.928412 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxwdn\" (UniqueName: \"kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn\") pod \"maas-api-revoke-sub-046e1343-6nsf4\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:24.051196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:24.051118 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:24.423196 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:24.423158 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4"] Nov 03 21:25:24.430030 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:25:24.429997 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84cffa2_637d_48f9_a560_32392c970eab.slice/crio-fee85d5316fb17fb68440d5d19a8189774cad44fc134d2f3f6e77bb457221d1f WatchSource:0}: Error finding container fee85d5316fb17fb68440d5d19a8189774cad44fc134d2f3f6e77bb457221d1f: Status 404 returned error can't find the container with id fee85d5316fb17fb68440d5d19a8189774cad44fc134d2f3f6e77bb457221d1f Nov 03 21:25:25.150122 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:25.150083 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" event={"ID":"e84cffa2-637d-48f9-a560-32392c970eab","Type":"ContainerStarted","Data":"5a112d3f5206e029524f3cc9c189e47840e72d961b1302e308d9ba10204c5f45"} Nov 03 21:25:25.150614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:25.150170 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" event={"ID":"e84cffa2-637d-48f9-a560-32392c970eab","Type":"ContainerStarted","Data":"fee85d5316fb17fb68440d5d19a8189774cad44fc134d2f3f6e77bb457221d1f"} Nov 03 21:25:25.260788 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:25.260726 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" podStartSLOduration=2.260710993 podStartE2EDuration="2.260710993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:25:25.259386389 +0000 UTC m=+3333.324627862" watchObservedRunningTime="2026-11-03 21:25:25.260710993 +0000 UTC m=+3333.325952445" Nov 03 21:25:26.157789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:26.157749 5932 generic.go:356] "Generic (PLEG): container finished" podID="e84cffa2-637d-48f9-a560-32392c970eab" containerID="5a112d3f5206e029524f3cc9c189e47840e72d961b1302e308d9ba10204c5f45" exitCode=0 Nov 03 21:25:26.158190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:26.157811 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" event={"ID":"e84cffa2-637d-48f9-a560-32392c970eab","Type":"ContainerDied","Data":"5a112d3f5206e029524f3cc9c189e47840e72d961b1302e308d9ba10204c5f45"} Nov 03 21:25:27.296826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.296797 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:27.300960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.300938 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca\") pod \"e84cffa2-637d-48f9-a560-32392c970eab\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " Nov 03 21:25:27.301098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.301035 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca\") pod \"e84cffa2-637d-48f9-a560-32392c970eab\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " Nov 03 21:25:27.301098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.301060 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn\" (UniqueName: \"kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn\") pod \"e84cffa2-637d-48f9-a560-32392c970eab\" (UID: \"e84cffa2-637d-48f9-a560-32392c970eab\") " Nov 03 21:25:27.301338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.301313 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca" pod "e84cffa2-637d-48f9-a560-32392c970eab" (UID: "e84cffa2-637d-48f9-a560-32392c970eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:27.303249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.303223 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn" pod "e84cffa2-637d-48f9-a560-32392c970eab" (UID: "e84cffa2-637d-48f9-a560-32392c970eab"). InnerVolumeSpecName "kube-api-access-vxwdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:27.303340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.303313 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca" pod "e84cffa2-637d-48f9-a560-32392c970eab" (UID: "e84cffa2-637d-48f9-a560-32392c970eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:27.402659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.402601 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:27.402659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.402648 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxwdn\" (UniqueName: \"kubernetes.io/projected/e84cffa2-637d-48f9-a560-32392c970eab-kube-api-access-vxwdn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:27.402659 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:27.402662 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84cffa2-637d-48f9-a560-32392c970eab-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:28.171761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:28.171729 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" Nov 03 21:25:28.171975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:28.171759 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4" event={"ID":"e84cffa2-637d-48f9-a560-32392c970eab","Type":"ContainerDied","Data":"fee85d5316fb17fb68440d5d19a8189774cad44fc134d2f3f6e77bb457221d1f"} Nov 03 21:25:28.171975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:28.171807 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee85d5316fb17fb68440d5d19a8189774cad44fc134d2f3f6e77bb457221d1f" Nov 03 21:25:34.608062 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.607971 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq"] Nov 03 21:25:34.608636 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.608602 5932 cpu_manager.go:414] "containerMap: removing container" podUID="e84cffa2-637d-48f9-a560-32392c970eab" containerName="revoke-keys" Nov 03 21:25:34.608770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.608653 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84cffa2-637d-48f9-a560-32392c970eab" containerName="revoke-keys" Nov 03 21:25:34.608770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.608752 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84cffa2-637d-48f9-a560-32392c970eab" containerName="revoke-keys" Nov 03 21:25:34.612284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.612265 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.675100 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.675059 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.675266 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.675229 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdrvk\" (UniqueName: \"kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.675333 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.675267 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.709200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.709162 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:25:34.709382 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.709201 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:25:34.711607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.711583 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq"] Nov 03 21:25:34.776661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.776599 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.776860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.776719 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdrvk\" (UniqueName: \"kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.776860 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.776746 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.777299 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.777280 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.779120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.779102 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.834501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.834468 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdrvk\" (UniqueName: \"kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk\") pod \"maas-api-revoke-sub-d3fba056-b59qq\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:34.925338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:34.925308 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:35.297707 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:35.297589 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq"] Nov 03 21:25:35.303847 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:25:35.303802 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edb4149_afb6_4d08_add6_78a7b99b70df.slice/crio-9442431b304cb08ff280e498f4f30984ce197f5687fa1c726c28a4da6c82236a WatchSource:0}: Error finding container 9442431b304cb08ff280e498f4f30984ce197f5687fa1c726c28a4da6c82236a: Status 404 returned error can't find the container with id 9442431b304cb08ff280e498f4f30984ce197f5687fa1c726c28a4da6c82236a Nov 03 21:25:36.220675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:36.220641 5932 generic.go:356] "Generic (PLEG): container finished" podID="1edb4149-afb6-4d08-add6-78a7b99b70df" containerID="647e97aabb0bae3ddba5d760f3a063a5b252a2e6913dfb5d4d6608f186786f75" exitCode=0 Nov 03 21:25:36.220675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:36.220682 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" event={"ID":"1edb4149-afb6-4d08-add6-78a7b99b70df","Type":"ContainerDied","Data":"647e97aabb0bae3ddba5d760f3a063a5b252a2e6913dfb5d4d6608f186786f75"} Nov 03 21:25:36.221124 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:36.220734 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" event={"ID":"1edb4149-afb6-4d08-add6-78a7b99b70df","Type":"ContainerStarted","Data":"9442431b304cb08ff280e498f4f30984ce197f5687fa1c726c28a4da6c82236a"} Nov 03 21:25:37.435775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.435742 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:37.505505 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.505389 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca\") pod \"1edb4149-afb6-4d08-add6-78a7b99b70df\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " Nov 03 21:25:37.505755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.505509 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca\") pod \"1edb4149-afb6-4d08-add6-78a7b99b70df\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " Nov 03 21:25:37.505755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.505674 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk\" (UniqueName: \"kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk\") pod \"1edb4149-afb6-4d08-add6-78a7b99b70df\" (UID: \"1edb4149-afb6-4d08-add6-78a7b99b70df\") " Nov 03 21:25:37.505917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.505850 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca" pod "1edb4149-afb6-4d08-add6-78a7b99b70df" (UID: "1edb4149-afb6-4d08-add6-78a7b99b70df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:25:37.506085 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.506061 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:37.507812 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.507761 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca" pod "1edb4149-afb6-4d08-add6-78a7b99b70df" (UID: "1edb4149-afb6-4d08-add6-78a7b99b70df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:25:37.508226 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.508195 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk" pod "1edb4149-afb6-4d08-add6-78a7b99b70df" (UID: "1edb4149-afb6-4d08-add6-78a7b99b70df"). InnerVolumeSpecName "kube-api-access-xdrvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:25:37.607645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.607589 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1edb4149-afb6-4d08-add6-78a7b99b70df-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:37.607645 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:37.607648 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdrvk\" (UniqueName: \"kubernetes.io/projected/1edb4149-afb6-4d08-add6-78a7b99b70df-kube-api-access-xdrvk\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:25:38.272648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:38.272165 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" event={"ID":"1edb4149-afb6-4d08-add6-78a7b99b70df","Type":"ContainerDied","Data":"9442431b304cb08ff280e498f4f30984ce197f5687fa1c726c28a4da6c82236a"} Nov 03 21:25:38.272648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:38.272303 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9442431b304cb08ff280e498f4f30984ce197f5687fa1c726c28a4da6c82236a" Nov 03 21:25:38.272648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:25:38.272400 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq" Nov 03 21:25:44.150696 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:25:44.150646 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:26:01.126829 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:01.126791 5932 scope.go:122] "RemoveContainer" containerID="47b3d0be22e0992c112ee75a4afc63123adb91c088de42132a2be8095ed2a642" Nov 03 21:26:01.136846 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:01.136823 5932 scope.go:122] "RemoveContainer" containerID="134314d64eca1c1f6b4f8ede4f2a38befd7e86d28aa2dc5e1aae337af907b565" Nov 03 21:26:13.427454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.427419 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w"] Nov 03 21:26:13.427894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.427880 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1edb4149-afb6-4d08-add6-78a7b99b70df" containerName="revoke-keys" Nov 03 21:26:13.427894 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.427895 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1edb4149-afb6-4d08-add6-78a7b99b70df" containerName="revoke-keys" Nov 03 21:26:13.428001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.427991 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1edb4149-afb6-4d08-add6-78a7b99b70df" containerName="revoke-keys" Nov 03 21:26:13.432917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.432888 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.483190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.483129 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:26:13.529530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.529493 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:26:13.532103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.532073 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w"] Nov 03 21:26:13.574260 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.574218 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.574469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.574281 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.574469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.574367 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbqf\" (UniqueName: \"kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.675826 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.675790 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.676057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.675865 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.676057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.675894 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbqf\" (UniqueName: \"kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.676610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.676582 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.678224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.678174 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.733614 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.733572 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbqf\" (UniqueName: \"kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf\") pod \"maas-api-revoke-sub-ae594979-jf82w\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:13.746457 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:13.746422 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:14.132033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:14.131995 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w"] Nov 03 21:26:14.136003 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:26:14.135967 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1f1a26_0591_4656_bf64_b8097c8a4ca5.slice/crio-02ece656c9273df256d9775b9bc49cb9c9404e1d69ee6798e7dc685ffd4b71bf WatchSource:0}: Error finding container 02ece656c9273df256d9775b9bc49cb9c9404e1d69ee6798e7dc685ffd4b71bf: Status 404 returned error can't find the container with id 02ece656c9273df256d9775b9bc49cb9c9404e1d69ee6798e7dc685ffd4b71bf Nov 03 21:26:14.154218 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:26:14.154186 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:26:14.565218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:14.565182 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" event={"ID":"ac1f1a26-0591-4656-bf64-b8097c8a4ca5","Type":"ContainerStarted","Data":"559b3b5ac09eb448432d4648adda6a8642687187bd7d19bd3b552ac3ca789b9e"} Nov 03 21:26:14.565673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:14.565241 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" event={"ID":"ac1f1a26-0591-4656-bf64-b8097c8a4ca5","Type":"ContainerStarted","Data":"02ece656c9273df256d9775b9bc49cb9c9404e1d69ee6798e7dc685ffd4b71bf"} Nov 03 21:26:14.676033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:14.675974 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" podStartSLOduration=1.675957265 podStartE2EDuration="1.675957265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:14.673749794 +0000 UTC m=+3382.738991268" watchObservedRunningTime="2026-11-03 21:26:14.675957265 +0000 UTC m=+3382.741198738" Nov 03 21:26:15.572557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:15.572525 5932 generic.go:356] "Generic (PLEG): container finished" podID="ac1f1a26-0591-4656-bf64-b8097c8a4ca5" containerID="559b3b5ac09eb448432d4648adda6a8642687187bd7d19bd3b552ac3ca789b9e" exitCode=0 Nov 03 21:26:15.572989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:15.572610 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" event={"ID":"ac1f1a26-0591-4656-bf64-b8097c8a4ca5","Type":"ContainerDied","Data":"559b3b5ac09eb448432d4648adda6a8642687187bd7d19bd3b552ac3ca789b9e"} Nov 03 21:26:16.725800 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.725765 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:16.809436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.809395 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf\" (UniqueName: \"kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf\") pod \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " Nov 03 21:26:16.809691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.809507 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca\") pod \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " Nov 03 21:26:16.809691 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.809584 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca\") pod \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\" (UID: \"ac1f1a26-0591-4656-bf64-b8097c8a4ca5\") " Nov 03 21:26:16.809981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.809947 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca" pod "ac1f1a26-0591-4656-bf64-b8097c8a4ca5" (UID: "ac1f1a26-0591-4656-bf64-b8097c8a4ca5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:16.811756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.811726 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf" pod "ac1f1a26-0591-4656-bf64-b8097c8a4ca5" (UID: "ac1f1a26-0591-4656-bf64-b8097c8a4ca5"). InnerVolumeSpecName "kube-api-access-8zbqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:16.811756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.811742 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca" pod "ac1f1a26-0591-4656-bf64-b8097c8a4ca5" (UID: "ac1f1a26-0591-4656-bf64-b8097c8a4ca5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:16.910494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.910455 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbqf\" (UniqueName: \"kubernetes.io/projected/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-kube-api-access-8zbqf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:16.910494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.910490 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:16.910494 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:16.910503 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac1f1a26-0591-4656-bf64-b8097c8a4ca5-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:17.586754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:17.586722 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" Nov 03 21:26:17.586938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:17.586722 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w" event={"ID":"ac1f1a26-0591-4656-bf64-b8097c8a4ca5","Type":"ContainerDied","Data":"02ece656c9273df256d9775b9bc49cb9c9404e1d69ee6798e7dc685ffd4b71bf"} Nov 03 21:26:17.586938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:17.586871 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ece656c9273df256d9775b9bc49cb9c9404e1d69ee6798e7dc685ffd4b71bf" Nov 03 21:26:42.055063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:42.055029 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2"] Nov 03 21:26:42.100979 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:42.100941 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f01949aa-qhwb2"] Nov 03 21:26:44.051131 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:44.051094 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ef152d-2e94-417d-ae48-f9dbfd1088e7" path="/var/lib/kubelet/pods/b2ef152d-2e94-417d-ae48-f9dbfd1088e7/volumes" Nov 03 21:26:44.151423 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:26:44.151378 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:26:44.944407 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:44.944375 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt"] Nov 03 21:26:44.944852 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:44.944837 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ac1f1a26-0591-4656-bf64-b8097c8a4ca5" containerName="revoke-keys" Nov 03 21:26:44.944907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:44.944854 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1f1a26-0591-4656-bf64-b8097c8a4ca5" containerName="revoke-keys" Nov 03 21:26:44.944952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:44.944943 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1f1a26-0591-4656-bf64-b8097c8a4ca5" containerName="revoke-keys" Nov 03 21:26:44.948296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:44.948278 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.002355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.002313 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt"] Nov 03 21:26:45.045193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.045156 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:26:45.045376 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.045254 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:26:45.137056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.137014 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcpms\" (UniqueName: \"kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.137526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.137070 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.137526 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.137235 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.238856 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.238754 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.239057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.238866 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcpms\" (UniqueName: \"kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.239057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.238892 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.239564 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.239541 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.241445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.241413 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.297087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.297050 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcpms\" (UniqueName: \"kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms\") pod \"maas-api-revoke-sub-a4aef7d7-xtqrt\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.561925 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.561836 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:45.937667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:45.937615 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt"] Nov 03 21:26:45.943466 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:26:45.943436 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230c7ffb_5136_4838_88c6_8d048d62726d.slice/crio-50a98016f23733e410b5d0a6afe5e3c35ce50b77bd4033c0afd33d0bfb48e6b1 WatchSource:0}: Error finding container 50a98016f23733e410b5d0a6afe5e3c35ce50b77bd4033c0afd33d0bfb48e6b1: Status 404 returned error can't find the container with id 50a98016f23733e410b5d0a6afe5e3c35ce50b77bd4033c0afd33d0bfb48e6b1 Nov 03 21:26:46.821674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:46.821609 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" event={"ID":"230c7ffb-5136-4838-88c6-8d048d62726d","Type":"ContainerStarted","Data":"c2152c39762788480c63d2997f9caa7e38f308752eb75654ffe066e5ad74ddda"} Nov 03 21:26:46.822066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:46.821691 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" event={"ID":"230c7ffb-5136-4838-88c6-8d048d62726d","Type":"ContainerStarted","Data":"50a98016f23733e410b5d0a6afe5e3c35ce50b77bd4033c0afd33d0bfb48e6b1"} Nov 03 21:26:46.933306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:46.933247 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" podStartSLOduration=2.933232861 podStartE2EDuration="2.933232861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:26:46.930884078 +0000 UTC m=+3414.996125553" watchObservedRunningTime="2026-11-03 21:26:46.933232861 +0000 UTC m=+3414.998474333" Nov 03 21:26:47.828987 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:47.828949 5932 generic.go:356] "Generic (PLEG): container finished" podID="230c7ffb-5136-4838-88c6-8d048d62726d" containerID="c2152c39762788480c63d2997f9caa7e38f308752eb75654ffe066e5ad74ddda" exitCode=0 Nov 03 21:26:47.829551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:47.828993 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" event={"ID":"230c7ffb-5136-4838-88c6-8d048d62726d","Type":"ContainerDied","Data":"c2152c39762788480c63d2997f9caa7e38f308752eb75654ffe066e5ad74ddda"} Nov 03 21:26:48.976437 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:48.976403 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:49.073613 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.073575 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca\") pod \"230c7ffb-5136-4838-88c6-8d048d62726d\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " Nov 03 21:26:49.073843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.073682 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca\") pod \"230c7ffb-5136-4838-88c6-8d048d62726d\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " Nov 03 21:26:49.073843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.073717 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms\" (UniqueName: \"kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms\") pod \"230c7ffb-5136-4838-88c6-8d048d62726d\" (UID: \"230c7ffb-5136-4838-88c6-8d048d62726d\") " Nov 03 21:26:49.074084 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.074055 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca" pod "230c7ffb-5136-4838-88c6-8d048d62726d" (UID: "230c7ffb-5136-4838-88c6-8d048d62726d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:26:49.075813 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.075774 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca" pod "230c7ffb-5136-4838-88c6-8d048d62726d" (UID: "230c7ffb-5136-4838-88c6-8d048d62726d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:26:49.075921 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.075891 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms" pod "230c7ffb-5136-4838-88c6-8d048d62726d" (UID: "230c7ffb-5136-4838-88c6-8d048d62726d"). InnerVolumeSpecName "kube-api-access-kcpms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:26:49.175406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.175366 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:49.175406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.175400 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230c7ffb-5136-4838-88c6-8d048d62726d-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:49.175406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.175410 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcpms\" (UniqueName: \"kubernetes.io/projected/230c7ffb-5136-4838-88c6-8d048d62726d-kube-api-access-kcpms\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:26:49.850697 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.850651 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" event={"ID":"230c7ffb-5136-4838-88c6-8d048d62726d","Type":"ContainerDied","Data":"50a98016f23733e410b5d0a6afe5e3c35ce50b77bd4033c0afd33d0bfb48e6b1"} Nov 03 21:26:49.850931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.850720 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt" Nov 03 21:26:49.850931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:26:49.850749 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a98016f23733e410b5d0a6afe5e3c35ce50b77bd4033c0afd33d0bfb48e6b1" Nov 03 21:27:01.167040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:01.167007 5932 scope.go:122] "RemoveContainer" containerID="4184e9a23bcf3ed5b2520d56c4a4f6d4fbdb4ed04113410dd591f3f66433a3b5" Nov 03 21:27:14.150546 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:27:14.150457 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:27:28.803501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.803461 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z"] Nov 03 21:27:28.804292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.803971 5932 cpu_manager.go:414] "containerMap: removing container" podUID="230c7ffb-5136-4838-88c6-8d048d62726d" containerName="revoke-keys" Nov 03 21:27:28.804292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.803985 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230c7ffb-5136-4838-88c6-8d048d62726d" containerName="revoke-keys" Nov 03 21:27:28.804292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.804082 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="230c7ffb-5136-4838-88c6-8d048d62726d" containerName="revoke-keys" Nov 03 21:27:28.807144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.807120 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:28.861109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.861076 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z"] Nov 03 21:27:28.870479 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.870453 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:27:28.870658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.870540 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:27:28.915446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.915394 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:28.915671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.915463 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8mjn\" (UniqueName: \"kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:28.915671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:28.915547 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.016205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.016161 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.016430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.016217 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8mjn\" (UniqueName: \"kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.016430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.016241 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.016994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.016971 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.018675 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.018652 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.074140 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.074061 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8mjn\" (UniqueName: \"kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn\") pod \"maas-api-revoke-sub-79805ca9-rc88z\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.121189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.121154 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:29.498245 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:29.498205 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z"] Nov 03 21:27:29.503870 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:27:29.503835 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7392bb3b_4012_4694_aef2_b5f4227227ff.slice/crio-948fd16db0ee77ff70397573e105d39698e609ad8cbef124ab4ef61a38ae3231 WatchSource:0}: Error finding container 948fd16db0ee77ff70397573e105d39698e609ad8cbef124ab4ef61a38ae3231: Status 404 returned error can't find the container with id 948fd16db0ee77ff70397573e105d39698e609ad8cbef124ab4ef61a38ae3231 Nov 03 21:27:30.140252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:30.140216 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" event={"ID":"7392bb3b-4012-4694-aef2-b5f4227227ff","Type":"ContainerStarted","Data":"a465a0ae927a410b4ec7479c922493b241ed40b5fcdaae94f451d252cab4bac0"} Nov 03 21:27:30.140669 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:30.140269 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" event={"ID":"7392bb3b-4012-4694-aef2-b5f4227227ff","Type":"ContainerStarted","Data":"948fd16db0ee77ff70397573e105d39698e609ad8cbef124ab4ef61a38ae3231"} Nov 03 21:27:30.250778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:30.250716 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" podStartSLOduration=2.25070017 podStartE2EDuration="2.25070017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:27:30.248375039 +0000 UTC m=+3458.313616513" watchObservedRunningTime="2026-11-03 21:27:30.25070017 +0000 UTC m=+3458.315941649" Nov 03 21:27:31.147235 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:31.147198 5932 generic.go:356] "Generic (PLEG): container finished" podID="7392bb3b-4012-4694-aef2-b5f4227227ff" containerID="a465a0ae927a410b4ec7479c922493b241ed40b5fcdaae94f451d252cab4bac0" exitCode=0 Nov 03 21:27:31.147710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:31.147280 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" event={"ID":"7392bb3b-4012-4694-aef2-b5f4227227ff","Type":"ContainerDied","Data":"a465a0ae927a410b4ec7479c922493b241ed40b5fcdaae94f451d252cab4bac0"} Nov 03 21:27:32.300476 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.300448 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:32.453450 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.453360 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca\") pod \"7392bb3b-4012-4694-aef2-b5f4227227ff\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " Nov 03 21:27:32.453607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.453526 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca\") pod \"7392bb3b-4012-4694-aef2-b5f4227227ff\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " Nov 03 21:27:32.453607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.453562 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn\" (UniqueName: \"kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn\") pod \"7392bb3b-4012-4694-aef2-b5f4227227ff\" (UID: \"7392bb3b-4012-4694-aef2-b5f4227227ff\") " Nov 03 21:27:32.453929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.453897 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca" pod "7392bb3b-4012-4694-aef2-b5f4227227ff" (UID: "7392bb3b-4012-4694-aef2-b5f4227227ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:32.455597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.455573 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca" pod "7392bb3b-4012-4694-aef2-b5f4227227ff" (UID: "7392bb3b-4012-4694-aef2-b5f4227227ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:32.455743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.455641 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn" pod "7392bb3b-4012-4694-aef2-b5f4227227ff" (UID: "7392bb3b-4012-4694-aef2-b5f4227227ff"). InnerVolumeSpecName "kube-api-access-r8mjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:32.554350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.554316 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:32.554350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.554344 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8mjn\" (UniqueName: \"kubernetes.io/projected/7392bb3b-4012-4694-aef2-b5f4227227ff-kube-api-access-r8mjn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:32.554350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:32.554354 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7392bb3b-4012-4694-aef2-b5f4227227ff-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:33.160404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:33.160375 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" Nov 03 21:27:33.160594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:33.160408 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z" event={"ID":"7392bb3b-4012-4694-aef2-b5f4227227ff","Type":"ContainerDied","Data":"948fd16db0ee77ff70397573e105d39698e609ad8cbef124ab4ef61a38ae3231"} Nov 03 21:27:33.160594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:33.160452 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="948fd16db0ee77ff70397573e105d39698e609ad8cbef124ab4ef61a38ae3231" Nov 03 21:27:35.046892 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:35.046829 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq"] Nov 03 21:27:35.093223 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:35.093188 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-64cxq"] Nov 03 21:27:36.050887 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:36.050853 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e547156d-b6e6-4b8e-9e34-2faeeba8ea1e" path="/var/lib/kubelet/pods/e547156d-b6e6-4b8e-9e34-2faeeba8ea1e/volumes" Nov 03 21:27:44.157725 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:27:44.157677 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:27:52.675843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.675758 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg"] Nov 03 21:27:52.676314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.676224 5932 cpu_manager.go:414] "containerMap: removing container" podUID="7392bb3b-4012-4694-aef2-b5f4227227ff" containerName="revoke-keys" Nov 03 21:27:52.676314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.676235 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7392bb3b-4012-4694-aef2-b5f4227227ff" containerName="revoke-keys" Nov 03 21:27:52.676314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.676309 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="7392bb3b-4012-4694-aef2-b5f4227227ff" containerName="revoke-keys" Nov 03 21:27:52.679759 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.679743 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.735161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.735128 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg"] Nov 03 21:27:52.763608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.763563 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.763852 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.763697 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.763852 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.763751 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8c4j\" (UniqueName: \"kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.777035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.777001 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:27:52.777189 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.777044 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:27:52.864445 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.864399 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8c4j\" (UniqueName: \"kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.864612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.864524 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.864612 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.864600 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.865268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.865244 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.867164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.867127 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.922763 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.922732 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8c4j\" (UniqueName: \"kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j\") pod \"maas-api-revoke-sub-0b7e8edb-8n6lg\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:52.992328 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:52.992235 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:53.370358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:53.370277 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg"] Nov 03 21:27:53.375964 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:27:53.375936 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e0f0c6_4db9_4c59_978d_add9940575d9.slice/crio-daaf41a4418d59db369c7283e22dbd48ac85ebb417c5c28fdcf1a51e3e80a451 WatchSource:0}: Error finding container daaf41a4418d59db369c7283e22dbd48ac85ebb417c5c28fdcf1a51e3e80a451: Status 404 returned error can't find the container with id daaf41a4418d59db369c7283e22dbd48ac85ebb417c5c28fdcf1a51e3e80a451 Nov 03 21:27:54.323415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:54.323382 5932 generic.go:356] "Generic (PLEG): container finished" podID="a5e0f0c6-4db9-4c59-978d-add9940575d9" containerID="8d6d8fd5b6a7d6c6f9c229929e67777a89607cbfb8bd70bfc080c3f4fb21bdae" exitCode=0 Nov 03 21:27:54.323824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:54.323443 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" event={"ID":"a5e0f0c6-4db9-4c59-978d-add9940575d9","Type":"ContainerDied","Data":"8d6d8fd5b6a7d6c6f9c229929e67777a89607cbfb8bd70bfc080c3f4fb21bdae"} Nov 03 21:27:54.323824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:54.323479 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" event={"ID":"a5e0f0c6-4db9-4c59-978d-add9940575d9","Type":"ContainerStarted","Data":"daaf41a4418d59db369c7283e22dbd48ac85ebb417c5c28fdcf1a51e3e80a451"} Nov 03 21:27:55.465635 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.465599 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:55.489951 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.489926 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca\") pod \"a5e0f0c6-4db9-4c59-978d-add9940575d9\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " Nov 03 21:27:55.490068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.490052 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca\") pod \"a5e0f0c6-4db9-4c59-978d-add9940575d9\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " Nov 03 21:27:55.490160 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.490145 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j\" (UniqueName: \"kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j\") pod \"a5e0f0c6-4db9-4c59-978d-add9940575d9\" (UID: \"a5e0f0c6-4db9-4c59-978d-add9940575d9\") " Nov 03 21:27:55.490456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.490430 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca" pod "a5e0f0c6-4db9-4c59-978d-add9940575d9" (UID: "a5e0f0c6-4db9-4c59-978d-add9940575d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:27:55.491966 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.491936 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca" pod "a5e0f0c6-4db9-4c59-978d-add9940575d9" (UID: "a5e0f0c6-4db9-4c59-978d-add9940575d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:27:55.492204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.492182 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j" pod "a5e0f0c6-4db9-4c59-978d-add9940575d9" (UID: "a5e0f0c6-4db9-4c59-978d-add9940575d9"). InnerVolumeSpecName "kube-api-access-j8c4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:27:55.592054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.591968 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:55.592054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.591996 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8c4j\" (UniqueName: \"kubernetes.io/projected/a5e0f0c6-4db9-4c59-978d-add9940575d9-kube-api-access-j8c4j\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:55.592054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:55.592006 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5e0f0c6-4db9-4c59-978d-add9940575d9-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:27:56.337863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:56.337774 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" Nov 03 21:27:56.337863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:56.337811 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg" event={"ID":"a5e0f0c6-4db9-4c59-978d-add9940575d9","Type":"ContainerDied","Data":"daaf41a4418d59db369c7283e22dbd48ac85ebb417c5c28fdcf1a51e3e80a451"} Nov 03 21:27:56.338069 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:27:56.337873 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daaf41a4418d59db369c7283e22dbd48ac85ebb417c5c28fdcf1a51e3e80a451" Nov 03 21:28:01.191120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:01.191087 5932 scope.go:122] "RemoveContainer" containerID="b4a4944c6cd0f3655cd139860e0cf3823f992c7710ef4651a69d494fdeb9328e" Nov 03 21:28:01.201310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:01.201289 5932 scope.go:122] "RemoveContainer" containerID="ac7ed39bf02573d6d5412422884e6b9a08279c66161b61e0f0582200223414bc" Nov 03 21:28:04.080981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.080946 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6"] Nov 03 21:28:04.081521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.081383 5932 cpu_manager.go:414] "containerMap: removing container" podUID="a5e0f0c6-4db9-4c59-978d-add9940575d9" containerName="revoke-keys" Nov 03 21:28:04.081521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.081394 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e0f0c6-4db9-4c59-978d-add9940575d9" containerName="revoke-keys" Nov 03 21:28:04.081521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.081492 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e0f0c6-4db9-4c59-978d-add9940575d9" containerName="revoke-keys" Nov 03 21:28:04.086439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.086394 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.135658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.135563 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:04.178705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.178667 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.178898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.178792 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn74c\" (UniqueName: \"kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.178898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.178867 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.181552 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.181534 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:28:04.184281 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.184257 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6"] Nov 03 21:28:04.280090 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.280053 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn74c\" (UniqueName: \"kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.280330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.280103 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.280330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.280179 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.280827 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.280805 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.282576 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.282545 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.337451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.337368 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn74c\" (UniqueName: \"kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c\") pod \"maas-api-revoke-sub-3d4156f1-nnxk6\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.399779 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.399749 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:04.776964 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:04.776933 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6"] Nov 03 21:28:04.783665 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:28:04.783592 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89384784_cf41_4380_a898_7d8f58660e48.slice/crio-2903810cb1ec5a490afd73b8380da5cc312c613d9be9160dcfa5169ce23e18d1 WatchSource:0}: Error finding container 2903810cb1ec5a490afd73b8380da5cc312c613d9be9160dcfa5169ce23e18d1: Status 404 returned error can't find the container with id 2903810cb1ec5a490afd73b8380da5cc312c613d9be9160dcfa5169ce23e18d1 Nov 03 21:28:05.393599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:05.393565 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" event={"ID":"89384784-cf41-4380-a898-7d8f58660e48","Type":"ContainerStarted","Data":"48d7dbdc8a5d2e01d3d9c8c9278aa00e464e679736fabd4f82b491bfea821604"} Nov 03 21:28:05.394094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:05.393616 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" event={"ID":"89384784-cf41-4380-a898-7d8f58660e48","Type":"ContainerStarted","Data":"2903810cb1ec5a490afd73b8380da5cc312c613d9be9160dcfa5169ce23e18d1"} Nov 03 21:28:05.502673 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:05.502597 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" podStartSLOduration=1.5025811340000002 podStartE2EDuration="1.502581134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:05.501848589 +0000 UTC m=+3493.567090063" watchObservedRunningTime="2026-11-03 21:28:05.502581134 +0000 UTC m=+3493.567822607" Nov 03 21:28:06.411537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:06.411498 5932 generic.go:356] "Generic (PLEG): container finished" podID="89384784-cf41-4380-a898-7d8f58660e48" containerID="48d7dbdc8a5d2e01d3d9c8c9278aa00e464e679736fabd4f82b491bfea821604" exitCode=0 Nov 03 21:28:06.412345 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:06.411563 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" event={"ID":"89384784-cf41-4380-a898-7d8f58660e48","Type":"ContainerDied","Data":"48d7dbdc8a5d2e01d3d9c8c9278aa00e464e679736fabd4f82b491bfea821604"} Nov 03 21:28:07.745217 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.744852 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:07.820849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.820809 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c\" (UniqueName: \"kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c\") pod \"89384784-cf41-4380-a898-7d8f58660e48\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " Nov 03 21:28:07.821068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.820883 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca\") pod \"89384784-cf41-4380-a898-7d8f58660e48\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " Nov 03 21:28:07.821068 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.820918 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca\") pod \"89384784-cf41-4380-a898-7d8f58660e48\" (UID: \"89384784-cf41-4380-a898-7d8f58660e48\") " Nov 03 21:28:07.821218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.821190 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca" pod "89384784-cf41-4380-a898-7d8f58660e48" (UID: "89384784-cf41-4380-a898-7d8f58660e48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:07.822985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.822958 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca" pod "89384784-cf41-4380-a898-7d8f58660e48" (UID: "89384784-cf41-4380-a898-7d8f58660e48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:07.823101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.823056 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c" pod "89384784-cf41-4380-a898-7d8f58660e48" (UID: "89384784-cf41-4380-a898-7d8f58660e48"). InnerVolumeSpecName "kube-api-access-bn74c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:07.922555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.922507 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn74c\" (UniqueName: \"kubernetes.io/projected/89384784-cf41-4380-a898-7d8f58660e48-kube-api-access-bn74c\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:07.922555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.922543 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89384784-cf41-4380-a898-7d8f58660e48-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:07.922555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:07.922562 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89384784-cf41-4380-a898-7d8f58660e48-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:08.424862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:08.424835 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" Nov 03 21:28:08.425055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:08.424834 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6" event={"ID":"89384784-cf41-4380-a898-7d8f58660e48","Type":"ContainerDied","Data":"2903810cb1ec5a490afd73b8380da5cc312c613d9be9160dcfa5169ce23e18d1"} Nov 03 21:28:08.425055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:08.424969 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2903810cb1ec5a490afd73b8380da5cc312c613d9be9160dcfa5169ce23e18d1" Nov 03 21:28:14.151053 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:28:14.151016 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:28:37.931165 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:37.931080 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6"] Nov 03 21:28:37.977369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:37.977336 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-nnxk6"] Nov 03 21:28:38.050751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.050714 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89384784-cf41-4380-a898-7d8f58660e48" path="/var/lib/kubelet/pods/89384784-cf41-4380-a898-7d8f58660e48/volumes" Nov 03 21:28:38.055440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.055415 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd"] Nov 03 21:28:38.055878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.055865 5932 cpu_manager.go:414] "containerMap: removing container" podUID="89384784-cf41-4380-a898-7d8f58660e48" containerName="revoke-keys" Nov 03 21:28:38.055937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.055879 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89384784-cf41-4380-a898-7d8f58660e48" containerName="revoke-keys" Nov 03 21:28:38.055981 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.055950 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="89384784-cf41-4380-a898-7d8f58660e48" containerName="revoke-keys" Nov 03 21:28:38.059354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.059338 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.113706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.113676 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd"] Nov 03 21:28:38.131888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.131856 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:28:38.132048 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.131854 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:28:38.151646 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.151606 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.151804 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.151680 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9q86\" (UniqueName: \"kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.151804 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.151772 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.252319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.252239 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.252471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.252317 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9q86\" (UniqueName: \"kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.252471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.252366 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.253116 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.253090 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.254674 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.254655 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.309562 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.309525 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9q86\" (UniqueName: \"kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86\") pod \"maas-api-revoke-sub-3d4156f1-l2bxd\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.372198 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.372168 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:38.745005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:38.744974 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd"] Nov 03 21:28:38.751043 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:28:38.751012 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2a1edc_f682_4806_9023_b767e58fb35a.slice/crio-2c8e48714871fdf3bb0af7ed921880ad810ca88131604bb5c50ebd1568eb6546 WatchSource:0}: Error finding container 2c8e48714871fdf3bb0af7ed921880ad810ca88131604bb5c50ebd1568eb6546: Status 404 returned error can't find the container with id 2c8e48714871fdf3bb0af7ed921880ad810ca88131604bb5c50ebd1568eb6546 Nov 03 21:28:39.615427 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:39.615393 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" event={"ID":"4e2a1edc-f682-4806-9023-b767e58fb35a","Type":"ContainerStarted","Data":"5d42c32dbde06485a9cfb1cb64fb10382b5cd825e597fc40402752105300edc3"} Nov 03 21:28:39.615834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:39.615493 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" event={"ID":"4e2a1edc-f682-4806-9023-b767e58fb35a","Type":"ContainerStarted","Data":"2c8e48714871fdf3bb0af7ed921880ad810ca88131604bb5c50ebd1568eb6546"} Nov 03 21:28:39.724006 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:39.723951 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" podStartSLOduration=1.723933242 podStartE2EDuration="1.723933242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:28:39.723141942 +0000 UTC m=+3527.788383415" watchObservedRunningTime="2026-11-03 21:28:39.723933242 +0000 UTC m=+3527.789174715" Nov 03 21:28:40.623913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:40.623875 5932 generic.go:356] "Generic (PLEG): container finished" podID="4e2a1edc-f682-4806-9023-b767e58fb35a" containerID="5d42c32dbde06485a9cfb1cb64fb10382b5cd825e597fc40402752105300edc3" exitCode=0 Nov 03 21:28:40.624312 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:40.623966 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" event={"ID":"4e2a1edc-f682-4806-9023-b767e58fb35a","Type":"ContainerDied","Data":"5d42c32dbde06485a9cfb1cb64fb10382b5cd825e597fc40402752105300edc3"} Nov 03 21:28:41.764234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.764207 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:41.787336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.787304 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86\" (UniqueName: \"kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86\") pod \"4e2a1edc-f682-4806-9023-b767e58fb35a\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " Nov 03 21:28:41.787469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.787427 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca\") pod \"4e2a1edc-f682-4806-9023-b767e58fb35a\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " Nov 03 21:28:41.787514 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.787491 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca\") pod \"4e2a1edc-f682-4806-9023-b767e58fb35a\" (UID: \"4e2a1edc-f682-4806-9023-b767e58fb35a\") " Nov 03 21:28:41.787856 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.787832 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca" pod "4e2a1edc-f682-4806-9023-b767e58fb35a" (UID: "4e2a1edc-f682-4806-9023-b767e58fb35a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:28:41.789583 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.789562 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca" pod "4e2a1edc-f682-4806-9023-b767e58fb35a" (UID: "4e2a1edc-f682-4806-9023-b767e58fb35a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:28:41.789689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.789665 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86" pod "4e2a1edc-f682-4806-9023-b767e58fb35a" (UID: "4e2a1edc-f682-4806-9023-b767e58fb35a"). InnerVolumeSpecName "kube-api-access-b9q86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:28:41.888972 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.888694 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:41.888972 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.888726 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e2a1edc-f682-4806-9023-b767e58fb35a-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:41.888972 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:41.888741 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9q86\" (UniqueName: \"kubernetes.io/projected/4e2a1edc-f682-4806-9023-b767e58fb35a-kube-api-access-b9q86\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:28:42.637530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:42.637497 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" Nov 03 21:28:42.637530 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:42.637500 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd" event={"ID":"4e2a1edc-f682-4806-9023-b767e58fb35a","Type":"ContainerDied","Data":"2c8e48714871fdf3bb0af7ed921880ad810ca88131604bb5c50ebd1568eb6546"} Nov 03 21:28:42.637783 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:42.637556 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c8e48714871fdf3bb0af7ed921880ad810ca88131604bb5c50ebd1568eb6546" Nov 03 21:28:44.151436 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:28:44.151398 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:28:45.055610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:45.055576 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v"] Nov 03 21:28:45.101761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:45.101724 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8ll7v"] Nov 03 21:28:46.050089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:28:46.050044 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3589b691-f94d-4d56-af28-1423b5993834" path="/var/lib/kubelet/pods/3589b691-f94d-4d56-af28-1423b5993834/volumes" Nov 03 21:29:01.231653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:01.231608 5932 scope.go:122] "RemoveContainer" containerID="dfec9e15f6a6f534fbcb723c45c1bca788664ebc010103ff5c6f27cd4130a5e1" Nov 03 21:29:14.151541 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:29:14.151503 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:29:16.056915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:16.056878 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq"] Nov 03 21:29:16.103692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:16.103656 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-fvvsq"] Nov 03 21:29:18.051441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:18.051401 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd0e2a8-430d-4308-9581-d91ce348706b" path="/var/lib/kubelet/pods/ccd0e2a8-430d-4308-9581-d91ce348706b/volumes" Nov 03 21:29:29.045307 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.045271 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz"] Nov 03 21:29:29.090975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.090932 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-q94gz"] Nov 03 21:29:29.283980 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.283947 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7"] Nov 03 21:29:29.284442 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.284428 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4e2a1edc-f682-4806-9023-b767e58fb35a" containerName="revoke-keys" Nov 03 21:29:29.284498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.284444 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e2a1edc-f682-4806-9023-b767e58fb35a" containerName="revoke-keys" Nov 03 21:29:29.284546 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.284535 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e2a1edc-f682-4806-9023-b767e58fb35a" containerName="revoke-keys" Nov 03 21:29:29.287870 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.287846 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.339513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.339420 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:29:29.339849 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.339776 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:29:29.342616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.342587 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7"] Nov 03 21:29:29.410784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.410752 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.410982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.410792 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.410982 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.410874 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c45mh\" (UniqueName: \"kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.512377 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.512339 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.512377 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.512383 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.512717 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.512406 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c45mh\" (UniqueName: \"kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.513144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.513118 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.514871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.514851 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.570252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.570213 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c45mh\" (UniqueName: \"kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh\") pod \"maas-api-revoke-sub-e2fc9c5b-x2mm7\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.601493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.601405 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:29.975968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:29.975930 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7"] Nov 03 21:29:29.981838 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:29:29.981806 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dbb0a8_7a7b_4e61_8a2d_9288ca42394f.slice/crio-d8d1c7c66c3ef1cf2c5b591d5d1a53907b58a92b805490e58650d773a77a2ffb WatchSource:0}: Error finding container d8d1c7c66c3ef1cf2c5b591d5d1a53907b58a92b805490e58650d773a77a2ffb: Status 404 returned error can't find the container with id d8d1c7c66c3ef1cf2c5b591d5d1a53907b58a92b805490e58650d773a77a2ffb Nov 03 21:29:30.050726 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:30.050694 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2685431-119c-4a64-b40c-99dae2f04a6c" path="/var/lib/kubelet/pods/e2685431-119c-4a64-b40c-99dae2f04a6c/volumes" Nov 03 21:29:30.957751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:30.957717 5932 generic.go:356] "Generic (PLEG): container finished" podID="87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" containerID="9892d0e10edd70864cb250fae4c5eaf465f5d50b7b2ce187af1d9f73ef483f7c" exitCode=0 Nov 03 21:29:30.957933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:30.957807 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" event={"ID":"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f","Type":"ContainerDied","Data":"9892d0e10edd70864cb250fae4c5eaf465f5d50b7b2ce187af1d9f73ef483f7c"} Nov 03 21:29:30.957933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:30.957865 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" event={"ID":"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f","Type":"ContainerStarted","Data":"d8d1c7c66c3ef1cf2c5b591d5d1a53907b58a92b805490e58650d773a77a2ffb"} Nov 03 21:29:32.120145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.120118 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:32.241053 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.240964 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca\") pod \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " Nov 03 21:29:32.241212 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.241070 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh\" (UniqueName: \"kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh\") pod \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " Nov 03 21:29:32.241212 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.241157 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca\") pod \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\" (UID: \"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f\") " Nov 03 21:29:32.241762 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.241727 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca" pod "87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" (UID: "87dbb0a8-7a7b-4e61-8a2d-9288ca42394f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:32.243080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.243059 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca" pod "87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" (UID: "87dbb0a8-7a7b-4e61-8a2d-9288ca42394f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:32.243554 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.243538 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh" pod "87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" (UID: "87dbb0a8-7a7b-4e61-8a2d-9288ca42394f"). InnerVolumeSpecName "kube-api-access-c45mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:32.342121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.342095 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:32.342121 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.342123 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:32.342282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.342133 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c45mh\" (UniqueName: \"kubernetes.io/projected/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f-kube-api-access-c45mh\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:32.971072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.971041 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" Nov 03 21:29:32.971251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.971067 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7" event={"ID":"87dbb0a8-7a7b-4e61-8a2d-9288ca42394f","Type":"ContainerDied","Data":"d8d1c7c66c3ef1cf2c5b591d5d1a53907b58a92b805490e58650d773a77a2ffb"} Nov 03 21:29:32.971251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:32.971116 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8d1c7c66c3ef1cf2c5b591d5d1a53907b58a92b805490e58650d773a77a2ffb" Nov 03 21:29:40.054823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.054783 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff"] Nov 03 21:29:40.055255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.055241 5932 cpu_manager.go:414] "containerMap: removing container" podUID="87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" containerName="revoke-keys" Nov 03 21:29:40.055301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.055255 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" containerName="revoke-keys" Nov 03 21:29:40.055348 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.055339 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" containerName="revoke-keys" Nov 03 21:29:40.058773 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.058751 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.112527 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.112487 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff"] Nov 03 21:29:40.154953 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.154916 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:29:40.154953 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.154919 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:29:40.220207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.220168 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbm9b\" (UniqueName: \"kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.220446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.220233 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.220446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.220398 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.322145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.322046 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.322145 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.322127 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.322408 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.322182 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbm9b\" (UniqueName: \"kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.322734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.322709 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.324498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.324475 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.380172 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.380131 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbm9b\" (UniqueName: \"kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b\") pod \"maas-api-revoke-sub-5a29efb3-t62ff\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:40.673066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:40.673025 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:41.047922 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:41.047838 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff"] Nov 03 21:29:41.053559 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:29:41.053525 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a8fbb5_8fd7_40dc_8335_0e735dff3fc1.slice/crio-5675e89071898920af0bcc28670076cb5fd550954380d254dbbd623136bf4876 WatchSource:0}: Error finding container 5675e89071898920af0bcc28670076cb5fd550954380d254dbbd623136bf4876: Status 404 returned error can't find the container with id 5675e89071898920af0bcc28670076cb5fd550954380d254dbbd623136bf4876 Nov 03 21:29:42.027995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:42.027956 5932 generic.go:356] "Generic (PLEG): container finished" podID="43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" containerID="4cb5f4b55367b2ff9d75b64cfd0a3fbc128c88baaee841388f4eaf758b957bad" exitCode=0 Nov 03 21:29:42.028406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:42.028046 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" event={"ID":"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1","Type":"ContainerDied","Data":"4cb5f4b55367b2ff9d75b64cfd0a3fbc128c88baaee841388f4eaf758b957bad"} Nov 03 21:29:42.028406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:42.028120 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" event={"ID":"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1","Type":"ContainerStarted","Data":"5675e89071898920af0bcc28670076cb5fd550954380d254dbbd623136bf4876"} Nov 03 21:29:43.218293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.217840 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:43.355926 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.355830 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca\") pod \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " Nov 03 21:29:43.355926 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.355909 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca\") pod \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " Nov 03 21:29:43.356174 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.355997 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b\" (UniqueName: \"kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b\") pod \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\" (UID: \"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1\") " Nov 03 21:29:43.358595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.358548 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca" pod "43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" (UID: "43a8fbb5-8fd7-40dc-8335-0e735dff3fc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:29:43.360030 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.359991 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b" pod "43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" (UID: "43a8fbb5-8fd7-40dc-8335-0e735dff3fc1"). InnerVolumeSpecName "kube-api-access-hbm9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:29:43.360174 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.360114 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca" pod "43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" (UID: "43a8fbb5-8fd7-40dc-8335-0e735dff3fc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:29:43.457551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.457486 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbm9b\" (UniqueName: \"kubernetes.io/projected/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-kube-api-access-hbm9b\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:43.457551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.457545 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:43.457551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:43.457559 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:29:44.065422 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:44.065388 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" Nov 03 21:29:44.097927 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:44.097791 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff" event={"ID":"43a8fbb5-8fd7-40dc-8335-0e735dff3fc1","Type":"ContainerDied","Data":"5675e89071898920af0bcc28670076cb5fd550954380d254dbbd623136bf4876"} Nov 03 21:29:44.098264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:44.098194 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5675e89071898920af0bcc28670076cb5fd550954380d254dbbd623136bf4876" Nov 03 21:29:44.217384 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:29:44.217340 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:29:56.284935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:56.284898 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:29:56.302301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:56.302269 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:29:58.997587 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:58.997552 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd"] Nov 03 21:29:59.044255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.044224 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-l2bxd"] Nov 03 21:29:59.121706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.121653 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw"] Nov 03 21:29:59.122298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.122263 5932 cpu_manager.go:414] "containerMap: removing container" podUID="43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" containerName="revoke-keys" Nov 03 21:29:59.122298 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.122288 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" containerName="revoke-keys" Nov 03 21:29:59.122444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.122429 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" containerName="revoke-keys" Nov 03 21:29:59.125857 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.125837 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.175902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.175870 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:29:59.176848 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.176803 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:29:59.180809 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.180763 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw"] Nov 03 21:29:59.257416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.257328 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.257416 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.257395 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6gn\" (UniqueName: \"kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.257665 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.257530 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.359094 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.359052 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.359378 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.359115 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.359378 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.359176 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6gn\" (UniqueName: \"kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.359957 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.359929 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.361594 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.361569 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.417291 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.417249 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6gn\" (UniqueName: \"kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn\") pod \"maas-api-revoke-sub-3d4156f1-hp6nw\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.440198 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.440157 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:29:59.816833 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:29:59.816798 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw"] Nov 03 21:29:59.822552 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:29:59.822516 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f3c0c5_57f5_48a8_a4e2_c1d40e26d8c5.slice/crio-cdf5ca25a757694b8436a4e61497ad311cced1b66eee63e2d551ac144dde0e3c WatchSource:0}: Error finding container cdf5ca25a757694b8436a4e61497ad311cced1b66eee63e2d551ac144dde0e3c: Status 404 returned error can't find the container with id cdf5ca25a757694b8436a4e61497ad311cced1b66eee63e2d551ac144dde0e3c Nov 03 21:30:00.052670 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:00.052616 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e2a1edc-f682-4806-9023-b767e58fb35a" path="/var/lib/kubelet/pods/4e2a1edc-f682-4806-9023-b767e58fb35a/volumes" Nov 03 21:30:00.189888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:00.189849 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" event={"ID":"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5","Type":"ContainerStarted","Data":"0adfad905223e8dc1f521e23daf3209d3130473aa9e25c3b0e2605e4b9e8f8a0"} Nov 03 21:30:00.190075 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:00.189905 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" event={"ID":"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5","Type":"ContainerStarted","Data":"cdf5ca25a757694b8436a4e61497ad311cced1b66eee63e2d551ac144dde0e3c"} Nov 03 21:30:00.299216 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:00.299162 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" podStartSLOduration=1.299147182 podStartE2EDuration="1.299147182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:00.298113468 +0000 UTC m=+3608.363354943" watchObservedRunningTime="2026-11-03 21:30:00.299147182 +0000 UTC m=+3608.364388654" Nov 03 21:30:01.198488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:01.198452 5932 generic.go:356] "Generic (PLEG): container finished" podID="69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" containerID="0adfad905223e8dc1f521e23daf3209d3130473aa9e25c3b0e2605e4b9e8f8a0" exitCode=0 Nov 03 21:30:01.198950 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:01.198544 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" event={"ID":"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5","Type":"ContainerDied","Data":"0adfad905223e8dc1f521e23daf3209d3130473aa9e25c3b0e2605e4b9e8f8a0"} Nov 03 21:30:01.256200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:01.256162 5932 scope.go:122] "RemoveContainer" containerID="0ff419c12a77bf18738991896fd110c6a91be04e77ff879c5992fbc6e5db7178" Nov 03 21:30:02.353872 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.353844 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:30:02.494327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.494243 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca\") pod \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " Nov 03 21:30:02.494327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.494291 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca\") pod \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " Nov 03 21:30:02.494572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.494535 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn\" (UniqueName: \"kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn\") pod \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\" (UID: \"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5\") " Nov 03 21:30:02.494751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.494726 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca" pod "69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" (UID: "69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:30:02.494971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.494952 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:02.496772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.496745 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn" pod "69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" (UID: "69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5"). InnerVolumeSpecName "kube-api-access-9c6gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:30:02.497038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.497017 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca" pod "69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" (UID: "69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:30:02.596498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.596460 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c6gn\" (UniqueName: \"kubernetes.io/projected/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-kube-api-access-9c6gn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:02.596498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:02.596495 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:30:03.211816 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:03.211778 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" Nov 03 21:30:03.212018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:03.211810 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw" event={"ID":"69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5","Type":"ContainerDied","Data":"cdf5ca25a757694b8436a4e61497ad311cced1b66eee63e2d551ac144dde0e3c"} Nov 03 21:30:03.212018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:03.211860 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdf5ca25a757694b8436a4e61497ad311cced1b66eee63e2d551ac144dde0e3c" Nov 03 21:30:14.151965 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:30:14.151927 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:30:28.051034 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:28.051001 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4"] Nov 03 21:30:28.091916 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:28.091878 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-6nsf4"] Nov 03 21:30:30.050578 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:30.050541 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84cffa2-637d-48f9-a560-32392c970eab" path="/var/lib/kubelet/pods/e84cffa2-637d-48f9-a560-32392c970eab/volumes" Nov 03 21:30:38.059834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:38.059797 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq"] Nov 03 21:30:38.105931 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:38.105885 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-b59qq"] Nov 03 21:30:39.564802 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.564766 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2"] Nov 03 21:30:39.566345 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.566312 5932 cpu_manager.go:414] "containerMap: removing container" podUID="69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" containerName="revoke-keys" Nov 03 21:30:39.566568 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.566541 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" containerName="revoke-keys" Nov 03 21:30:39.566781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.566758 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" containerName="revoke-keys" Nov 03 21:30:39.576734 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.576703 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.609081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.609041 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2"] Nov 03 21:30:39.651990 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.651952 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-357e69\"" Nov 03 21:30:39.652184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.651995 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-5056388c-dockercfg-dswqw\"" Nov 03 21:30:39.653382 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.653358 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsbgf\" (UniqueName: \"kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf\") pod \"payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.653453 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.653417 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.754193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.754151 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsbgf\" (UniqueName: \"kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf\") pod \"payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.754386 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.754227 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.755007 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.754987 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.812743 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.812707 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsbgf\" (UniqueName: \"kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf\") pod \"payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:39.894553 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:39.894519 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:40.050314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:40.050278 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1edb4149-afb6-4d08-add6-78a7b99b70df" path="/var/lib/kubelet/pods/1edb4149-afb6-4d08-add6-78a7b99b70df/volumes" Nov 03 21:30:40.270599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:40.270499 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2"] Nov 03 21:30:40.275128 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:30:40.275096 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf8d3bc2_419a_4312_b293_633b9efbe490.slice/crio-fdd383f3342885c65ebdd421d108e9648ef343818ceb39a4aa47a09f94646110 WatchSource:0}: Error finding container fdd383f3342885c65ebdd421d108e9648ef343818ceb39a4aa47a09f94646110: Status 404 returned error can't find the container with id fdd383f3342885c65ebdd421d108e9648ef343818ceb39a4aa47a09f94646110 Nov 03 21:30:40.465666 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:40.465616 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" event={"ID":"cf8d3bc2-419a-4312-b293-633b9efbe490","Type":"ContainerStarted","Data":"ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90"} Nov 03 21:30:40.465862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:40.465685 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" event={"ID":"cf8d3bc2-419a-4312-b293-633b9efbe490","Type":"ContainerStarted","Data":"fdd383f3342885c65ebdd421d108e9648ef343818ceb39a4aa47a09f94646110"} Nov 03 21:30:40.465862 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:40.465844 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:30:40.576756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:40.576612 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" podStartSLOduration=1.576595707 podStartE2EDuration="1.576595707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:40.575462835 +0000 UTC m=+3648.640704308" watchObservedRunningTime="2026-11-03 21:30:40.576595707 +0000 UTC m=+3648.641837180" Nov 03 21:30:42.096561 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.096515 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w"] Nov 03 21:30:42.101249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.101220 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.179502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.179464 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.179502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.179508 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t7nx\" (UniqueName: \"kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.179834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.179809 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.179952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.179938 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.197410 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.197377 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-357e69-serving-cert\"" Nov 03 21:30:42.197574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.197444 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:30:42.197574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.197526 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\"" Nov 03 21:30:42.200290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.200265 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w"] Nov 03 21:30:42.280978 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.280939 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.281215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.281089 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.281215 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:30:42.281103 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-serving-cert: secret "maas-api-e2e-worker-single-main-357e69-serving-cert" not found Nov 03 21:30:42.281215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.281115 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t7nx\" (UniqueName: \"kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.281215 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.281146 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.281215 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:30:42.281175 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls podName:ad70f6d2-f2c6-4d2c-b00c-78ca332a219b nodeName:}" failed. No retries permitted until 2026-11-03 21:30:42.781152751 +0000 UTC m=+3650.846394203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls") pod "maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" (UID: "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b") : secret "maas-api-e2e-worker-single-main-357e69-serving-cert" not found Nov 03 21:30:42.281878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.281853 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.283898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.283866 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.368693 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.367860 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t7nx\" (UniqueName: \"kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.785678 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.785632 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:42.788213 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:42.788174 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:43.015834 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:43.015794 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:43.397211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:43.397172 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w"] Nov 03 21:30:43.402913 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:30:43.402882 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad70f6d2_f2c6_4d2c_b00c_78ca332a219b.slice/crio-4eb499cae314415e0b65dcbca12a27ea3e1bfb327670c2af5062bfc0117d1b07 WatchSource:0}: Error finding container 4eb499cae314415e0b65dcbca12a27ea3e1bfb327670c2af5062bfc0117d1b07: Status 404 returned error can't find the container with id 4eb499cae314415e0b65dcbca12a27ea3e1bfb327670c2af5062bfc0117d1b07 Nov 03 21:30:43.404510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:43.404487 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:30:43.485418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:43.485380 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" event={"ID":"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b","Type":"ContainerStarted","Data":"4eb499cae314415e0b65dcbca12a27ea3e1bfb327670c2af5062bfc0117d1b07"} Nov 03 21:30:44.150958 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:30:44.150919 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:30:44.493383 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:44.493300 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" event={"ID":"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b","Type":"ContainerStarted","Data":"1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108"} Nov 03 21:30:44.493782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:44.493386 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:44.604234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:44.604180 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" podStartSLOduration=1.674944583 podStartE2EDuration="2.604165969s" totalImagesPullingTime="929.221386ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:30:44.603320036 +0000 UTC m=+3652.668561533" watchObservedRunningTime="2026-11-03 21:30:44.604165969 +0000 UTC m=+3652.669407442" Nov 03 21:30:50.509186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:50.509153 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:30:51.476055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:30:51.476026 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:31:01.285200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:01.285162 5932 scope.go:122] "RemoveContainer" containerID="f6010881cc6baf76384b3002b5ee199bbf2a5ffa0d8abd9dc848f151f106fc56" Nov 03 21:31:01.297731 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:01.297697 5932 scope.go:122] "RemoveContainer" containerID="3d59f6c51a169226237ffcadd468f4d84b4e772519a42f279201ee46dd543d96" Nov 03 21:31:14.151089 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:31:14.151045 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:31:17.046463 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:17.046432 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w"] Nov 03 21:31:17.092488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:17.092456 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-jf82w"] Nov 03 21:31:18.051822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:18.051783 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1f1a26-0591-4656-bf64-b8097c8a4ca5" path="/var/lib/kubelet/pods/ac1f1a26-0591-4656-bf64-b8097c8a4ca5/volumes" Nov 03 21:31:44.151286 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:31:44.151207 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:31:50.051086 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:50.051054 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt"] Nov 03 21:31:50.091309 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:50.091264 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xtqrt"] Nov 03 21:31:52.050689 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:31:52.050658 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230c7ffb-5136-4838-88c6-8d048d62726d" path="/var/lib/kubelet/pods/230c7ffb-5136-4838-88c6-8d048d62726d/volumes" Nov 03 21:32:01.328360 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:01.328251 5932 scope.go:122] "RemoveContainer" containerID="5a112d3f5206e029524f3cc9c189e47840e72d961b1302e308d9ba10204c5f45" Nov 03 21:32:01.338200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:01.338174 5932 scope.go:122] "RemoveContainer" containerID="647e97aabb0bae3ddba5d760f3a063a5b252a2e6913dfb5d4d6608f186786f75" Nov 03 21:32:12.004954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:12.004911 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w"] Nov 03 21:32:12.005456 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:12.005152 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" podUID="ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" containerName="maas-api" containerID="cri-o://1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108" gracePeriod=30 Nov 03 21:32:12.170166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:12.170130 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2"] Nov 03 21:32:13.272700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.272672 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:32:13.363631 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.363532 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs\") pod \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " Nov 03 21:32:13.363631 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.363577 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\") pod \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " Nov 03 21:32:13.363631 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.363606 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca\") pod \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " Nov 03 21:32:13.363927 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.363699 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx\" (UniqueName: \"kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx\") pod \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\" (UID: \"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b\") " Nov 03 21:32:13.364018 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.363992 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca" pod "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" (UID: "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:13.365969 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.365935 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs" pod "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" (UID: "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:13.366091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.365966 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls" pod "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" (UID: "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:13.366091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.365940 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx" pod "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" (UID: "ad70f6d2-f2c6-4d2c-b00c-78ca332a219b"). InnerVolumeSpecName "kube-api-access-8t7nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:13.464945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.464905 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:13.464945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.464937 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:13.464945 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.464952 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:13.465199 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:13.464964 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t7nx\" (UniqueName: \"kubernetes.io/projected/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b-kube-api-access-8t7nx\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:14.070772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.070738 5932 generic.go:356] "Generic (PLEG): container finished" podID="ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" containerID="1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108" exitCode=0 Nov 03 21:32:14.070971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.070804 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" Nov 03 21:32:14.070971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.070830 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" event={"ID":"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b","Type":"ContainerDied","Data":"1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108"} Nov 03 21:32:14.070971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.070899 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w" event={"ID":"ad70f6d2-f2c6-4d2c-b00c-78ca332a219b","Type":"ContainerDied","Data":"4eb499cae314415e0b65dcbca12a27ea3e1bfb327670c2af5062bfc0117d1b07"} Nov 03 21:32:14.070971 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.070928 5932 scope.go:122] "RemoveContainer" containerID="1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108" Nov 03 21:32:14.081386 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.081363 5932 scope.go:122] "RemoveContainer" containerID="1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108" Nov 03 21:32:14.081734 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:32:14.081715 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108\": container with ID starting with 1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108 not found: ID does not exist" containerID="1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108" Nov 03 21:32:14.081824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.081753 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108"} err="failed to get container status \"1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108\": rpc error: code = NotFound desc = could not find container \"1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108\": container with ID starting with 1d597ab2d3e2171d625f00a9f9d9aa95cff572ced46dfee53d0733d68e9d5108 not found: ID does not exist" Nov 03 21:32:14.151988 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:32:14.151944 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:32:14.228104 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.228067 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w"] Nov 03 21:32:14.274234 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:14.274198 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-357e69-857b588c95-2kq5w"] Nov 03 21:32:16.050935 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:16.050903 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" path="/var/lib/kubelet/pods/ad70f6d2-f2c6-4d2c-b00c-78ca332a219b/volumes" Nov 03 21:32:17.170499 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:17.170417 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerName="payload-processing" containerID="cri-o://ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90" gracePeriod=25 Nov 03 21:32:21.474164 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:21.474125 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:32:25.777940 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.777902 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw"] Nov 03 21:32:25.824152 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.824116 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-hp6nw"] Nov 03 21:32:25.900718 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.900683 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc"] Nov 03 21:32:25.901157 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.901145 5932 cpu_manager.go:414] "containerMap: removing container" podUID="ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" containerName="maas-api" Nov 03 21:32:25.901218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.901159 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" containerName="maas-api" Nov 03 21:32:25.901278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.901259 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad70f6d2-f2c6-4d2c-b00c-78ca332a219b" containerName="maas-api" Nov 03 21:32:25.908710 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.907802 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:25.957708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.957673 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:32:25.957708 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.957672 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:32:25.959076 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:25.959046 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc"] Nov 03 21:32:26.004469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.004428 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8t9j\" (UniqueName: \"kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.004469 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.004477 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.004740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.004684 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.051330 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.051249 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5" path="/var/lib/kubelet/pods/69f3c0c5-57f5-48a8-a4e2-c1d40e26d8c5/volumes" Nov 03 21:32:26.106390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.106341 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.106680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.106429 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8t9j\" (UniqueName: \"kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.106680 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.106499 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.107523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.107458 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.110077 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.110037 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.165137 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.165100 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8t9j\" (UniqueName: \"kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j\") pod \"maas-api-revoke-sub-3d4156f1-r4qnc\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.223067 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.223028 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:26.596640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:26.596595 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc"] Nov 03 21:32:26.602969 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:32:26.602933 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0f3e18_9108_4631_b14a_72075a5407d1.slice/crio-156d4b0060b24a3169da6a6a472b5ed0fcc19dc877b92d0dc08e759accbe3239 WatchSource:0}: Error finding container 156d4b0060b24a3169da6a6a472b5ed0fcc19dc877b92d0dc08e759accbe3239: Status 404 returned error can't find the container with id 156d4b0060b24a3169da6a6a472b5ed0fcc19dc877b92d0dc08e759accbe3239 Nov 03 21:32:27.150404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:27.150365 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" event={"ID":"4f0f3e18-9108-4631-b14a-72075a5407d1","Type":"ContainerStarted","Data":"cd2c35daaaf0782ca9e83bca47ed5bbeeccf5fee37149ff49a37b28d2ab7372a"} Nov 03 21:32:27.150939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:27.150428 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" event={"ID":"4f0f3e18-9108-4631-b14a-72075a5407d1","Type":"ContainerStarted","Data":"156d4b0060b24a3169da6a6a472b5ed0fcc19dc877b92d0dc08e759accbe3239"} Nov 03 21:32:27.260424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:27.260327 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" podStartSLOduration=2.26031294 podStartE2EDuration="2.26031294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:32:27.258401208 +0000 UTC m=+3755.323642682" watchObservedRunningTime="2026-11-03 21:32:27.26031294 +0000 UTC m=+3755.325554413" Nov 03 21:32:28.158663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:28.158610 5932 generic.go:356] "Generic (PLEG): container finished" podID="4f0f3e18-9108-4631-b14a-72075a5407d1" containerID="cd2c35daaaf0782ca9e83bca47ed5bbeeccf5fee37149ff49a37b28d2ab7372a" exitCode=0 Nov 03 21:32:28.159074 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:28.158693 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" event={"ID":"4f0f3e18-9108-4631-b14a-72075a5407d1","Type":"ContainerDied","Data":"cd2c35daaaf0782ca9e83bca47ed5bbeeccf5fee37149ff49a37b28d2ab7372a"} Nov 03 21:32:29.309011 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.308987 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:29.445446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.445351 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j\" (UniqueName: \"kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j\") pod \"4f0f3e18-9108-4631-b14a-72075a5407d1\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " Nov 03 21:32:29.445446 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.445415 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca\") pod \"4f0f3e18-9108-4631-b14a-72075a5407d1\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " Nov 03 21:32:29.445778 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.445468 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca\") pod \"4f0f3e18-9108-4631-b14a-72075a5407d1\" (UID: \"4f0f3e18-9108-4631-b14a-72075a5407d1\") " Nov 03 21:32:29.445926 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.445899 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca" pod "4f0f3e18-9108-4631-b14a-72075a5407d1" (UID: "4f0f3e18-9108-4631-b14a-72075a5407d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:29.447722 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.447684 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca" pod "4f0f3e18-9108-4631-b14a-72075a5407d1" (UID: "4f0f3e18-9108-4631-b14a-72075a5407d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:32:29.447868 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.447778 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j" pod "4f0f3e18-9108-4631-b14a-72075a5407d1" (UID: "4f0f3e18-9108-4631-b14a-72075a5407d1"). InnerVolumeSpecName "kube-api-access-r8t9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:29.547412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.547367 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8t9j\" (UniqueName: \"kubernetes.io/projected/4f0f3e18-9108-4631-b14a-72075a5407d1-kube-api-access-r8t9j\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:29.547412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.547404 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:29.547412 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:29.547418 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f0f3e18-9108-4631-b14a-72075a5407d1-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:30.172839 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:30.172807 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" Nov 03 21:32:30.173019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:30.172841 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc" event={"ID":"4f0f3e18-9108-4631-b14a-72075a5407d1","Type":"ContainerDied","Data":"156d4b0060b24a3169da6a6a472b5ed0fcc19dc877b92d0dc08e759accbe3239"} Nov 03 21:32:30.173019 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:30.172899 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="156d4b0060b24a3169da6a6a472b5ed0fcc19dc877b92d0dc08e759accbe3239" Nov 03 21:32:31.473908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:31.473868 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:32:33.044696 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:33.044643 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z"] Nov 03 21:32:33.090897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:33.090860 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-rc88z"] Nov 03 21:32:34.050544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:34.050508 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7392bb3b-4012-4694-aef2-b5f4227227ff" path="/var/lib/kubelet/pods/7392bb3b-4012-4694-aef2-b5f4227227ff/volumes" Nov 03 21:32:37.432063 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.431997 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:32:37.525605 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.525571 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume\") pod \"cf8d3bc2-419a-4312-b293-633b9efbe490\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " Nov 03 21:32:37.525989 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.525964 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume" pod "cf8d3bc2-419a-4312-b293-633b9efbe490" (UID: "cf8d3bc2-419a-4312-b293-633b9efbe490"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:32:37.626109 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.626074 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf\" (UniqueName: \"kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf\") pod \"cf8d3bc2-419a-4312-b293-633b9efbe490\" (UID: \"cf8d3bc2-419a-4312-b293-633b9efbe490\") " Nov 03 21:32:37.626501 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.626481 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf8d3bc2-419a-4312-b293-633b9efbe490-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:37.628253 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.628230 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf" pod "cf8d3bc2-419a-4312-b293-633b9efbe490" (UID: "cf8d3bc2-419a-4312-b293-633b9efbe490"). InnerVolumeSpecName "kube-api-access-fsbgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:32:37.727856 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:37.727755 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsbgf\" (UniqueName: \"kubernetes.io/projected/cf8d3bc2-419a-4312-b293-633b9efbe490-kube-api-access-fsbgf\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:32:38.229467 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.229425 5932 generic.go:356] "Generic (PLEG): container finished" podID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerID="ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90" exitCode=0 Nov 03 21:32:38.229672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.229491 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" Nov 03 21:32:38.229672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.229519 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" event={"ID":"cf8d3bc2-419a-4312-b293-633b9efbe490","Type":"ContainerDied","Data":"ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90"} Nov 03 21:32:38.229672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.229604 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2" event={"ID":"cf8d3bc2-419a-4312-b293-633b9efbe490","Type":"ContainerDied","Data":"fdd383f3342885c65ebdd421d108e9648ef343818ceb39a4aa47a09f94646110"} Nov 03 21:32:38.229672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.229654 5932 scope.go:122] "RemoveContainer" containerID="ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90" Nov 03 21:32:38.239464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.239448 5932 scope.go:122] "RemoveContainer" containerID="ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90" Nov 03 21:32:38.239778 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:32:38.239747 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90\": container with ID starting with ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90 not found: ID does not exist" containerID="ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90" Nov 03 21:32:38.239847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.239788 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90"} err="failed to get container status \"ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90\": rpc error: code = NotFound desc = could not find container \"ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90\": container with ID starting with ea03bc0444b6a060e01d6e03989dfcd4baf0b0afc3d77b90fabc3837f2c2cd90 not found: ID does not exist" Nov 03 21:32:38.399252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.399212 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2"] Nov 03 21:32:38.404557 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:38.404528 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-357e69-57dd6bb85cqnl2"] Nov 03 21:32:40.050767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:40.050731 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" path="/var/lib/kubelet/pods/cf8d3bc2-419a-4312-b293-633b9efbe490/volumes" Nov 03 21:32:42.207181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:42.206788 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:32:42.207181 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:42.206930 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-45kq6" podUID="82930fab-d2aa-49e9-b83e-bcbc34c20ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:32:44.152435 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:32:44.152398 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:32:56.051438 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:56.051402 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg"] Nov 03 21:32:56.093996 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:56.093961 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-8n6lg"] Nov 03 21:32:58.050464 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:32:58.050425 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e0f0c6-4db9-4c59-978d-add9940575d9" path="/var/lib/kubelet/pods/a5e0f0c6-4db9-4c59-978d-add9940575d9/volumes" Nov 03 21:33:01.369595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:33:01.369557 5932 scope.go:122] "RemoveContainer" containerID="559b3b5ac09eb448432d4648adda6a8642687187bd7d19bd3b552ac3ca789b9e" Nov 03 21:33:01.379231 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:33:01.379208 5932 scope.go:122] "RemoveContainer" containerID="c2152c39762788480c63d2997f9caa7e38f308752eb75654ffe066e5ad74ddda" Nov 03 21:33:14.151361 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:33:14.151269 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:33:44.151817 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:33:44.151774 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:34:01.420301 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:01.420266 5932 scope.go:122] "RemoveContainer" containerID="8d6d8fd5b6a7d6c6f9c229929e67777a89607cbfb8bd70bfc080c3f4fb21bdae" Nov 03 21:34:01.430103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:01.430078 5932 scope.go:122] "RemoveContainer" containerID="a465a0ae927a410b4ec7479c922493b241ed40b5fcdaae94f451d252cab4bac0" Nov 03 21:34:14.154984 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:34:14.154940 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:34:26.731255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:26.730818 5932 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-lk747 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 21:34:26.731255 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:26.730904 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-lk747" podUID="0c400677-019d-4644-b45e-92057290a7cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 21:34:28.494750 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.494713 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6"] Nov 03 21:34:28.495170 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.495158 5932 cpu_manager.go:414] "containerMap: removing container" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerName="payload-processing" Nov 03 21:34:28.495214 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.495170 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerName="payload-processing" Nov 03 21:34:28.495214 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.495211 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4f0f3e18-9108-4631-b14a-72075a5407d1" containerName="revoke-keys" Nov 03 21:34:28.495321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.495217 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0f3e18-9108-4631-b14a-72075a5407d1" containerName="revoke-keys" Nov 03 21:34:28.495321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.495297 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8d3bc2-419a-4312-b293-633b9efbe490" containerName="payload-processing" Nov 03 21:34:28.495321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.495313 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0f3e18-9108-4631-b14a-72075a5407d1" containerName="revoke-keys" Nov 03 21:34:28.498522 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.498494 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.552134 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.552096 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6"] Nov 03 21:34:28.594751 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.594714 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:34:28.594934 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.594788 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:34:28.614296 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.614255 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.614459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.614408 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.614504 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.614461 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cfw\" (UniqueName: \"kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.715715 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.715671 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.715715 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.715721 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cfw\" (UniqueName: \"kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.715958 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.715791 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.716509 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.716488 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.718166 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.718146 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.773490 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.773412 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cfw\" (UniqueName: \"kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw\") pod \"maas-api-revoke-sub-d576e68b-r72d6\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:28.812300 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:28.812259 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:29.185764 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:29.185728 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6"] Nov 03 21:34:29.191942 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:34:29.191909 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccbef9f_d4e6_4744_925c_5289a2d2db5d.slice/crio-9fcc57e7a80332aa19d92523140a5b4f2c5756a9105e9526503fd41e51e84b71 WatchSource:0}: Error finding container 9fcc57e7a80332aa19d92523140a5b4f2c5756a9105e9526503fd41e51e84b71: Status 404 returned error can't find the container with id 9fcc57e7a80332aa19d92523140a5b4f2c5756a9105e9526503fd41e51e84b71 Nov 03 21:34:30.016775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:30.016734 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" event={"ID":"cccbef9f-d4e6-4744-925c-5289a2d2db5d","Type":"ContainerStarted","Data":"8c29a73c769fadf7022f632eed3ce10b03a05a1f53f798b3f3ef4f3d059984f1"} Nov 03 21:34:30.017284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:30.016798 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" event={"ID":"cccbef9f-d4e6-4744-925c-5289a2d2db5d","Type":"ContainerStarted","Data":"9fcc57e7a80332aa19d92523140a5b4f2c5756a9105e9526503fd41e51e84b71"} Nov 03 21:34:30.172347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:30.172287 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" podStartSLOduration=2.172274259 podStartE2EDuration="2.172274259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:34:30.124412367 +0000 UTC m=+3878.189653842" watchObservedRunningTime="2026-11-03 21:34:30.172274259 +0000 UTC m=+3878.237515770" Nov 03 21:34:31.023898 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:31.023866 5932 generic.go:356] "Generic (PLEG): container finished" podID="cccbef9f-d4e6-4744-925c-5289a2d2db5d" containerID="8c29a73c769fadf7022f632eed3ce10b03a05a1f53f798b3f3ef4f3d059984f1" exitCode=0 Nov 03 21:34:31.024297 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:31.023948 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" event={"ID":"cccbef9f-d4e6-4744-925c-5289a2d2db5d","Type":"ContainerDied","Data":"8c29a73c769fadf7022f632eed3ce10b03a05a1f53f798b3f3ef4f3d059984f1"} Nov 03 21:34:32.178917 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.178883 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:32.356426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.356332 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca\") pod \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " Nov 03 21:34:32.356617 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.356445 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw\" (UniqueName: \"kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw\") pod \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " Nov 03 21:34:32.356617 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.356470 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca\") pod \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\" (UID: \"cccbef9f-d4e6-4744-925c-5289a2d2db5d\") " Nov 03 21:34:32.356956 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.356929 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca" pod "cccbef9f-d4e6-4744-925c-5289a2d2db5d" (UID: "cccbef9f-d4e6-4744-925c-5289a2d2db5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:34:32.358661 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.358636 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca" pod "cccbef9f-d4e6-4744-925c-5289a2d2db5d" (UID: "cccbef9f-d4e6-4744-925c-5289a2d2db5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:34:32.358767 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.358673 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw" pod "cccbef9f-d4e6-4744-925c-5289a2d2db5d" (UID: "cccbef9f-d4e6-4744-925c-5289a2d2db5d"). InnerVolumeSpecName "kube-api-access-79cfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:34:32.457317 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.457277 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79cfw\" (UniqueName: \"kubernetes.io/projected/cccbef9f-d4e6-4744-925c-5289a2d2db5d-kube-api-access-79cfw\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:32.457317 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.457311 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:32.457317 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:32.457322 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cccbef9f-d4e6-4744-925c-5289a2d2db5d-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:34:33.037754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:33.037671 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" event={"ID":"cccbef9f-d4e6-4744-925c-5289a2d2db5d","Type":"ContainerDied","Data":"9fcc57e7a80332aa19d92523140a5b4f2c5756a9105e9526503fd41e51e84b71"} Nov 03 21:34:33.037754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:33.037723 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fcc57e7a80332aa19d92523140a5b4f2c5756a9105e9526503fd41e51e84b71" Nov 03 21:34:33.037754 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:33.037679 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6" Nov 03 21:34:33.049952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:33.049923 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7"] Nov 03 21:34:33.096589 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:33.096550 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-x2mm7"] Nov 03 21:34:34.051249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:34.051214 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87dbb0a8-7a7b-4e61-8a2d-9288ca42394f" path="/var/lib/kubelet/pods/87dbb0a8-7a7b-4e61-8a2d-9288ca42394f/volumes" Nov 03 21:34:44.050776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:44.050743 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff"] Nov 03 21:34:44.094911 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:44.094863 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-t62ff"] Nov 03 21:34:44.151748 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:34:44.151703 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:34:46.050141 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:46.050103 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a8fbb5-8fd7-40dc-8335-0e735dff3fc1" path="/var/lib/kubelet/pods/43a8fbb5-8fd7-40dc-8335-0e735dff3fc1/volumes" Nov 03 21:34:56.328803 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:56.328770 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:34:56.346126 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:56.346100 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:34:57.620742 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.620708 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22"] Nov 03 21:34:57.621271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.621255 5932 cpu_manager.go:414] "containerMap: removing container" podUID="cccbef9f-d4e6-4744-925c-5289a2d2db5d" containerName="revoke-keys" Nov 03 21:34:57.621324 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.621273 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccbef9f-d4e6-4744-925c-5289a2d2db5d" containerName="revoke-keys" Nov 03 21:34:57.621362 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.621352 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccbef9f-d4e6-4744-925c-5289a2d2db5d" containerName="revoke-keys" Nov 03 21:34:57.626252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.626226 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.675948 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.675677 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:34:57.721942 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.721903 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:34:57.724444 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.724417 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22"] Nov 03 21:34:57.772601 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.772557 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.772821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.772703 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.772821 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.772746 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tzt\" (UniqueName: \"kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.874108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.874007 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.874108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.874060 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tzt\" (UniqueName: \"kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.874378 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.874144 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.874873 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.874853 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.876551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.876526 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.932222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.932179 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tzt\" (UniqueName: \"kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt\") pod \"maas-api-revoke-sub-d2bc93bc-fnf22\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:57.939867 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:57.939823 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:34:58.316205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:58.316156 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22"] Nov 03 21:34:58.322483 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:34:58.322444 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4537cb2c_e53b_41f7_b53f_9d43eaf52233.slice/crio-3b0b71e89199fb59d8bda75cb5455bb3f7d08789ba35ea1bbae0a1d367aedb3f WatchSource:0}: Error finding container 3b0b71e89199fb59d8bda75cb5455bb3f7d08789ba35ea1bbae0a1d367aedb3f: Status 404 returned error can't find the container with id 3b0b71e89199fb59d8bda75cb5455bb3f7d08789ba35ea1bbae0a1d367aedb3f Nov 03 21:34:59.237899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:59.237869 5932 generic.go:356] "Generic (PLEG): container finished" podID="4537cb2c-e53b-41f7-b53f-9d43eaf52233" containerID="eb6e436e240814643c0b16da256975402636a99695f71069cef1fa48c8d4a2c0" exitCode=0 Nov 03 21:34:59.238282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:59.237950 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" event={"ID":"4537cb2c-e53b-41f7-b53f-9d43eaf52233","Type":"ContainerDied","Data":"eb6e436e240814643c0b16da256975402636a99695f71069cef1fa48c8d4a2c0"} Nov 03 21:34:59.238282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:34:59.238022 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" event={"ID":"4537cb2c-e53b-41f7-b53f-9d43eaf52233","Type":"ContainerStarted","Data":"3b0b71e89199fb59d8bda75cb5455bb3f7d08789ba35ea1bbae0a1d367aedb3f"} Nov 03 21:35:00.394500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.394473 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:35:00.500830 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.500730 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt\" (UniqueName: \"kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt\") pod \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " Nov 03 21:35:00.500830 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.500809 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca\") pod \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " Nov 03 21:35:00.501046 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.500909 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca\") pod \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\" (UID: \"4537cb2c-e53b-41f7-b53f-9d43eaf52233\") " Nov 03 21:35:00.501282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.501259 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca" pod "4537cb2c-e53b-41f7-b53f-9d43eaf52233" (UID: "4537cb2c-e53b-41f7-b53f-9d43eaf52233"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:35:00.503539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.503469 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca" pod "4537cb2c-e53b-41f7-b53f-9d43eaf52233" (UID: "4537cb2c-e53b-41f7-b53f-9d43eaf52233"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:35:00.503811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.503776 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt" pod "4537cb2c-e53b-41f7-b53f-9d43eaf52233" (UID: "4537cb2c-e53b-41f7-b53f-9d43eaf52233"). InnerVolumeSpecName "kube-api-access-x2tzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:35:00.602441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.602398 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tzt\" (UniqueName: \"kubernetes.io/projected/4537cb2c-e53b-41f7-b53f-9d43eaf52233-kube-api-access-x2tzt\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:00.602441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.602432 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:00.602441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:00.602442 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4537cb2c-e53b-41f7-b53f-9d43eaf52233-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:35:01.251064 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:01.251031 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" Nov 03 21:35:01.251251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:01.251070 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22" event={"ID":"4537cb2c-e53b-41f7-b53f-9d43eaf52233","Type":"ContainerDied","Data":"3b0b71e89199fb59d8bda75cb5455bb3f7d08789ba35ea1bbae0a1d367aedb3f"} Nov 03 21:35:01.251251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:01.251121 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b0b71e89199fb59d8bda75cb5455bb3f7d08789ba35ea1bbae0a1d367aedb3f" Nov 03 21:35:01.459877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:01.459840 5932 scope.go:122] "RemoveContainer" containerID="48d7dbdc8a5d2e01d3d9c8c9278aa00e464e679736fabd4f82b491bfea821604" Nov 03 21:35:01.470101 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:01.470074 5932 scope.go:122] "RemoveContainer" containerID="5d42c32dbde06485a9cfb1cb64fb10382b5cd825e597fc40402752105300edc3" Nov 03 21:35:14.152521 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:35:14.152479 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:35:18.818658 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.818604 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6"] Nov 03 21:35:18.819326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.819303 5932 cpu_manager.go:414] "containerMap: removing container" podUID="4537cb2c-e53b-41f7-b53f-9d43eaf52233" containerName="revoke-keys" Nov 03 21:35:18.819431 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.819328 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4537cb2c-e53b-41f7-b53f-9d43eaf52233" containerName="revoke-keys" Nov 03 21:35:18.819502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.819488 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="4537cb2c-e53b-41f7-b53f-9d43eaf52233" containerName="revoke-keys" Nov 03 21:35:18.824493 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.824463 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:18.878056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.878015 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6"] Nov 03 21:35:18.907700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.907611 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2x6\" (UniqueName: \"kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6\") pod \"payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:18.908161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.908133 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:18.924567 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.924530 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7b11dd\"" Nov 03 21:35:18.925060 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.925027 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e73c7e32-dockercfg-l5jp4\"" Nov 03 21:35:18.961744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.961703 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm"] Nov 03 21:35:18.966459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:18.966432 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.009070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.009033 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:19.009278 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.009119 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2x6\" (UniqueName: \"kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6\") pod \"payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:19.009838 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.009812 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:19.015684 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.015644 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm"] Nov 03 21:35:19.067581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.067539 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2x6\" (UniqueName: \"kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6\") pod \"payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:19.109879 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.109791 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.109879 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.109844 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxc5\" (UniqueName: \"kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5\") pod \"payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.138001 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.137967 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:19.210863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.210824 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.211015 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.210894 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxc5\" (UniqueName: \"kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5\") pod \"payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.212002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.211973 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.268447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.268409 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxc5\" (UniqueName: \"kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5\") pod \"payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.281119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.281082 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:19.510975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.510935 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6"] Nov 03 21:35:19.516548 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:35:19.516517 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d141ab_e8e1_4e64_8eb8_4b7d66e180ba.slice/crio-507763fc519afa4c5d4aa6bb01bce5a71da22b191e7847d49b8b63302aed37a1 WatchSource:0}: Error finding container 507763fc519afa4c5d4aa6bb01bce5a71da22b191e7847d49b8b63302aed37a1: Status 404 returned error can't find the container with id 507763fc519afa4c5d4aa6bb01bce5a71da22b191e7847d49b8b63302aed37a1 Nov 03 21:35:19.654276 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:19.654235 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm"] Nov 03 21:35:19.660554 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:35:19.660523 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79747de4_b8ec_407e_bd07_d9b8fa5b8a93.slice/crio-d1da99569837465a1c5aa287907d517842784f5473fae5545d24345941ff452a WatchSource:0}: Error finding container d1da99569837465a1c5aa287907d517842784f5473fae5545d24345941ff452a: Status 404 returned error can't find the container with id d1da99569837465a1c5aa287907d517842784f5473fae5545d24345941ff452a Nov 03 21:35:20.373755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.373717 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" event={"ID":"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba","Type":"ContainerStarted","Data":"f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4"} Nov 03 21:35:20.374289 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.373779 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" event={"ID":"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba","Type":"ContainerStarted","Data":"507763fc519afa4c5d4aa6bb01bce5a71da22b191e7847d49b8b63302aed37a1"} Nov 03 21:35:20.374289 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.373878 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:20.375503 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.375472 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" event={"ID":"79747de4-b8ec-407e-bd07-d9b8fa5b8a93","Type":"ContainerStarted","Data":"96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4"} Nov 03 21:35:20.375668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.375518 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:20.375668 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.375535 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" event={"ID":"79747de4-b8ec-407e-bd07-d9b8fa5b8a93","Type":"ContainerStarted","Data":"d1da99569837465a1c5aa287907d517842784f5473fae5545d24345941ff452a"} Nov 03 21:35:20.484460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.484397 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" podStartSLOduration=2.484379451 podStartE2EDuration="2.484379451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:20.481728269 +0000 UTC m=+3928.546969743" watchObservedRunningTime="2026-11-03 21:35:20.484379451 +0000 UTC m=+3928.549620968" Nov 03 21:35:20.606562 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:20.606253 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" podStartSLOduration=2.606233279 podStartE2EDuration="2.606233279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:20.589753648 +0000 UTC m=+3928.654995144" watchObservedRunningTime="2026-11-03 21:35:20.606233279 +0000 UTC m=+3928.671474751" Nov 03 21:35:21.493306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.493272 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6"] Nov 03 21:35:21.497270 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.497239 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.545962 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.545927 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-7b11dd-serving-cert\"" Nov 03 21:35:21.546204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.546084 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:35:21.592204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.592172 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\"" Nov 03 21:35:21.594844 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.594807 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6"] Nov 03 21:35:21.635024 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.634975 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czfkm\" (UniqueName: \"kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.635232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.635201 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.635306 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.635267 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.635411 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.635388 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.736004 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.735961 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czfkm\" (UniqueName: \"kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.736233 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.736071 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.736233 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.736102 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.736233 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.736129 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.736421 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:35:21.736239 5932 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-serving-cert: secret "maas-api-e2e-worker-single-main-7b11dd-serving-cert" not found Nov 03 21:35:21.736421 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:35:21.736292 5932 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls podName:c378aec5-9678-4034-bfab-eb2dbc1d7296 nodeName:}" failed. No retries permitted until 2026-11-03 21:35:22.236275777 +0000 UTC m=+3930.301517228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls") pod "maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" (UID: "c378aec5-9678-4034-bfab-eb2dbc1d7296") : secret "maas-api-e2e-worker-single-main-7b11dd-serving-cert" not found Nov 03 21:35:21.736947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.736922 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.741692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.738829 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:21.794054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:21.793967 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czfkm\" (UniqueName: \"kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:22.241033 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:22.240987 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:22.243544 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:22.243510 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:22.413958 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:22.413920 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:22.793379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:22.793344 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6"] Nov 03 21:35:22.799384 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:35:22.799350 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc378aec5_9678_4034_bfab_eb2dbc1d7296.slice/crio-0e0103ba807b97f9f94f14ed1541a5753f3c43550e37c39ebfec30e737cadbb6 WatchSource:0}: Error finding container 0e0103ba807b97f9f94f14ed1541a5753f3c43550e37c39ebfec30e737cadbb6: Status 404 returned error can't find the container with id 0e0103ba807b97f9f94f14ed1541a5753f3c43550e37c39ebfec30e737cadbb6 Nov 03 21:35:23.396272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:23.396225 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" event={"ID":"c378aec5-9678-4034-bfab-eb2dbc1d7296","Type":"ContainerStarted","Data":"0e0103ba807b97f9f94f14ed1541a5753f3c43550e37c39ebfec30e737cadbb6"} Nov 03 21:35:24.404713 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:24.404670 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" event={"ID":"c378aec5-9678-4034-bfab-eb2dbc1d7296","Type":"ContainerStarted","Data":"b7c6191acdbecaac861fa6cd2cb42d53071216557b3e114431d10a6084ef1659"} Nov 03 21:35:24.405220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:24.404759 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:24.515290 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:24.515220 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" podStartSLOduration=2.697225955 podStartE2EDuration="3.515205421s" totalImagesPullingTime="817.979466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:35:24.514406597 +0000 UTC m=+3932.579648071" watchObservedRunningTime="2026-11-03 21:35:24.515205421 +0000 UTC m=+3932.580446894" Nov 03 21:35:30.417358 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:30.417328 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:35:31.386366 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:31.386334 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:35:31.386569 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:35:31.386498 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:35:44.150890 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:35:44.150849 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:36:01.499825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:36:01.499788 5932 scope.go:122] "RemoveContainer" containerID="4cb5f4b55367b2ff9d75b64cfd0a3fbc128c88baaee841388f4eaf758b957bad" Nov 03 21:36:01.509579 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:36:01.509554 5932 scope.go:122] "RemoveContainer" containerID="0adfad905223e8dc1f521e23daf3209d3130473aa9e25c3b0e2605e4b9e8f8a0" Nov 03 21:36:01.519326 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:36:01.519300 5932 scope.go:122] "RemoveContainer" containerID="9892d0e10edd70864cb250fae4c5eaf465f5d50b7b2ce187af1d9f73ef483f7c" Nov 03 21:36:14.151813 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:36:14.151725 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:36:44.151117 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:36:44.151068 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:37:07.414111 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:07.414075 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm"] Nov 03 21:37:07.521899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:07.521863 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6"] Nov 03 21:37:12.414733 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:12.414686 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerName="payload-processing" containerID="cri-o://96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4" gracePeriod=25 Nov 03 21:37:12.522495 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:12.522432 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerName="payload-pre-processing" containerID="cri-o://f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4" gracePeriod=25 Nov 03 21:37:12.919197 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:12.919161 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6"] Nov 03 21:37:12.919394 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:12.919366 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" podUID="c378aec5-9678-4034-bfab-eb2dbc1d7296" containerName="maas-api" containerID="cri-o://b7c6191acdbecaac861fa6cd2cb42d53071216557b3e114431d10a6084ef1659" gracePeriod=30 Nov 03 21:37:14.070036 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.069985 5932 generic.go:356] "Generic (PLEG): container finished" podID="c378aec5-9678-4034-bfab-eb2dbc1d7296" containerID="b7c6191acdbecaac861fa6cd2cb42d53071216557b3e114431d10a6084ef1659" exitCode=0 Nov 03 21:37:14.070440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.070072 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" event={"ID":"c378aec5-9678-4034-bfab-eb2dbc1d7296","Type":"ContainerDied","Data":"b7c6191acdbecaac861fa6cd2cb42d53071216557b3e114431d10a6084ef1659"} Nov 03 21:37:14.159540 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:37:14.159480 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:37:14.192939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.192906 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:37:14.260607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.260517 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca\") pod \"c378aec5-9678-4034-bfab-eb2dbc1d7296\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " Nov 03 21:37:14.260607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.260559 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs\") pod \"c378aec5-9678-4034-bfab-eb2dbc1d7296\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " Nov 03 21:37:14.260863 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.260663 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\") pod \"c378aec5-9678-4034-bfab-eb2dbc1d7296\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " Nov 03 21:37:14.260994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.260968 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca" pod "c378aec5-9678-4034-bfab-eb2dbc1d7296" (UID: "c378aec5-9678-4034-bfab-eb2dbc1d7296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:37:14.262929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.262901 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs" pod "c378aec5-9678-4034-bfab-eb2dbc1d7296" (UID: "c378aec5-9678-4034-bfab-eb2dbc1d7296"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:37:14.263055 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.262931 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls" pod "c378aec5-9678-4034-bfab-eb2dbc1d7296" (UID: "c378aec5-9678-4034-bfab-eb2dbc1d7296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:37:14.361393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.361353 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm\" (UniqueName: \"kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm\") pod \"c378aec5-9678-4034-bfab-eb2dbc1d7296\" (UID: \"c378aec5-9678-4034-bfab-eb2dbc1d7296\") " Nov 03 21:37:14.361607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.361564 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c378aec5-9678-4034-bfab-eb2dbc1d7296-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:14.361607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.361579 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:14.361607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.361588 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c378aec5-9678-4034-bfab-eb2dbc1d7296-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:14.363663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.363616 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm" pod "c378aec5-9678-4034-bfab-eb2dbc1d7296" (UID: "c378aec5-9678-4034-bfab-eb2dbc1d7296"). InnerVolumeSpecName "kube-api-access-czfkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:37:14.462908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:14.462872 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czfkm\" (UniqueName: \"kubernetes.io/projected/c378aec5-9678-4034-bfab-eb2dbc1d7296-kube-api-access-czfkm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:15.078123 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:15.078091 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" Nov 03 21:37:15.078706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:15.078128 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6" event={"ID":"c378aec5-9678-4034-bfab-eb2dbc1d7296","Type":"ContainerDied","Data":"0e0103ba807b97f9f94f14ed1541a5753f3c43550e37c39ebfec30e737cadbb6"} Nov 03 21:37:15.078706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:15.078190 5932 scope.go:122] "RemoveContainer" containerID="b7c6191acdbecaac861fa6cd2cb42d53071216557b3e114431d10a6084ef1659" Nov 03 21:37:15.233744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:15.233707 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6"] Nov 03 21:37:15.279936 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:15.279851 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-7b11dd-7564769f77-772l6"] Nov 03 21:37:16.051428 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:16.051392 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c378aec5-9678-4034-bfab-eb2dbc1d7296" path="/var/lib/kubelet/pods/c378aec5-9678-4034-bfab-eb2dbc1d7296/volumes" Nov 03 21:37:21.384547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:21.384506 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:37:21.384547 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:21.384535 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:37:30.044237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:30.044189 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc"] Nov 03 21:37:30.090357 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:30.090319 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-r4qnc"] Nov 03 21:37:31.384295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:31.384254 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:37:31.384775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:31.384254 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 21:37:32.054837 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.052605 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0f3e18-9108-4631-b14a-72075a5407d1" path="/var/lib/kubelet/pods/4f0f3e18-9108-4631-b14a-72075a5407d1/volumes" Nov 03 21:37:32.686038 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.685970 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:37:32.751884 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.751837 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume\") pod \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " Nov 03 21:37:32.752096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.752046 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5\" (UniqueName: \"kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5\") pod \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\" (UID: \"79747de4-b8ec-407e-bd07-d9b8fa5b8a93\") " Nov 03 21:37:32.752486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.752289 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume" pod "79747de4-b8ec-407e-bd07-d9b8fa5b8a93" (UID: "79747de4-b8ec-407e-bd07-d9b8fa5b8a93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:37:32.752486 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.752420 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:32.755009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.754973 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5" pod "79747de4-b8ec-407e-bd07-d9b8fa5b8a93" (UID: "79747de4-b8ec-407e-bd07-d9b8fa5b8a93"). InnerVolumeSpecName "kube-api-access-pqxc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:37:32.774054 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.774025 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:37:32.852770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.852734 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6\" (UniqueName: \"kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6\") pod \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " Nov 03 21:37:32.852961 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.852805 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume\") pod \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\" (UID: \"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba\") " Nov 03 21:37:32.853115 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.853103 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqxc5\" (UniqueName: \"kubernetes.io/projected/79747de4-b8ec-407e-bd07-d9b8fa5b8a93-kube-api-access-pqxc5\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:32.853237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.853210 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume" pod "93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" (UID: "93d141ab-e8e1-4e64-8eb8-4b7d66e180ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:37:32.855065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.855043 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6" pod "93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" (UID: "93d141ab-e8e1-4e64-8eb8-4b7d66e180ba"). InnerVolumeSpecName "kube-api-access-zn2x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:37:32.954379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.954283 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2x6\" (UniqueName: \"kubernetes.io/projected/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-kube-api-access-zn2x6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:32.954379 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:32.954323 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:37:33.186179 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.186146 5932 generic.go:356] "Generic (PLEG): container finished" podID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerID="f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4" exitCode=0 Nov 03 21:37:33.186389 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.186198 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" Nov 03 21:37:33.186389 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.186227 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" event={"ID":"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba","Type":"ContainerDied","Data":"f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4"} Nov 03 21:37:33.186389 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.186296 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6" event={"ID":"93d141ab-e8e1-4e64-8eb8-4b7d66e180ba","Type":"ContainerDied","Data":"507763fc519afa4c5d4aa6bb01bce5a71da22b191e7847d49b8b63302aed37a1"} Nov 03 21:37:33.186389 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.186334 5932 scope.go:122] "RemoveContainer" containerID="f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4" Nov 03 21:37:33.187803 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.187770 5932 generic.go:356] "Generic (PLEG): container finished" podID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerID="96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4" exitCode=0 Nov 03 21:37:33.187954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.187819 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" Nov 03 21:37:33.187954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.187854 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" event={"ID":"79747de4-b8ec-407e-bd07-d9b8fa5b8a93","Type":"ContainerDied","Data":"96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4"} Nov 03 21:37:33.187954 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.187908 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm" event={"ID":"79747de4-b8ec-407e-bd07-d9b8fa5b8a93","Type":"ContainerDied","Data":"d1da99569837465a1c5aa287907d517842784f5473fae5545d24345941ff452a"} Nov 03 21:37:33.196434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.196409 5932 scope.go:122] "RemoveContainer" containerID="f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4" Nov 03 21:37:33.196850 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:37:33.196814 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4\": container with ID starting with f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4 not found: ID does not exist" containerID="f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4" Nov 03 21:37:33.196995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.196865 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4"} err="failed to get container status \"f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4\": rpc error: code = NotFound desc = could not find container \"f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4\": container with ID starting with f1198e2b9fa125e1bab24723876fae981d48e759cd55ee88cfa16312722d2bd4 not found: ID does not exist" Nov 03 21:37:33.196995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.196894 5932 scope.go:122] "RemoveContainer" containerID="96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4" Nov 03 21:37:33.206218 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.206196 5932 scope.go:122] "RemoveContainer" containerID="96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4" Nov 03 21:37:33.206528 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:37:33.206507 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4\": container with ID starting with 96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4 not found: ID does not exist" containerID="96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4" Nov 03 21:37:33.206584 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.206536 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4"} err="failed to get container status \"96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4\": rpc error: code = NotFound desc = could not find container \"96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4\": container with ID starting with 96b3bfffffa4b4a78b10afb609dffef693e80d92466db39aaf8c7e4598a84cf4 not found: ID does not exist" Nov 03 21:37:33.342207 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.342166 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm"] Nov 03 21:37:33.345915 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.345881 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7b11dd-66fbdb5d4w66mm"] Nov 03 21:37:33.447374 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.447337 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6"] Nov 03 21:37:33.451184 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:33.451153 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7b11dd-5b968hsnk6"] Nov 03 21:37:34.051349 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:34.051315 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" path="/var/lib/kubelet/pods/79747de4-b8ec-407e-bd07-d9b8fa5b8a93/volumes" Nov 03 21:37:34.051814 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:37:34.051796 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" path="/var/lib/kubelet/pods/93d141ab-e8e1-4e64-8eb8-4b7d66e180ba/volumes" Nov 03 21:37:44.151459 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:37:44.151420 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:38:14.214464 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:38:14.214425 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:38:41.892895 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.892862 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz"] Nov 03 21:38:41.893391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893310 5932 cpu_manager.go:414] "containerMap: removing container" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerName="payload-pre-processing" Nov 03 21:38:41.893391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893323 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerName="payload-pre-processing" Nov 03 21:38:41.893391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893337 5932 cpu_manager.go:414] "containerMap: removing container" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerName="payload-processing" Nov 03 21:38:41.893391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893342 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerName="payload-processing" Nov 03 21:38:41.893391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893355 5932 cpu_manager.go:414] "containerMap: removing container" podUID="c378aec5-9678-4034-bfab-eb2dbc1d7296" containerName="maas-api" Nov 03 21:38:41.893391 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893361 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c378aec5-9678-4034-bfab-eb2dbc1d7296" containerName="maas-api" Nov 03 21:38:41.893599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893426 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d141ab-e8e1-4e64-8eb8-4b7d66e180ba" containerName="payload-pre-processing" Nov 03 21:38:41.893599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893434 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="79747de4-b8ec-407e-bd07-d9b8fa5b8a93" containerName="payload-processing" Nov 03 21:38:41.893599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.893442 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="c378aec5-9678-4034-bfab-eb2dbc1d7296" containerName="maas-api" Nov 03 21:38:41.896683 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.896660 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:41.951401 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.951365 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:38:41.989163 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.989133 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:41.989163 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.989168 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5q7\" (UniqueName: \"kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:41.989388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.989294 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:41.997458 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.997434 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:38:41.999667 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:41.999647 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz"] Nov 03 21:38:42.090311 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.090274 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.090311 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.090310 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5q7\" (UniqueName: \"kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.090536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.090375 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.091040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.091015 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.092740 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.092717 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.148150 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.148072 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5q7\" (UniqueName: \"kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7\") pod \"maas-api-revoke-sub-452b6d7b-lswwz\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.211042 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.211001 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:42.588308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.588221 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz"] Nov 03 21:38:42.594246 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:38:42.594206 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8dcafa_92a8_4bcd_9ffd_01b2e69513c8.slice/crio-1a71c665219c27dfefa003572f89a99f29d3c748fe9735d5d7040af32d00fc57 WatchSource:0}: Error finding container 1a71c665219c27dfefa003572f89a99f29d3c748fe9735d5d7040af32d00fc57: Status 404 returned error can't find the container with id 1a71c665219c27dfefa003572f89a99f29d3c748fe9735d5d7040af32d00fc57 Nov 03 21:38:42.659040 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:42.658995 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" event={"ID":"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8","Type":"ContainerStarted","Data":"1a71c665219c27dfefa003572f89a99f29d3c748fe9735d5d7040af32d00fc57"} Nov 03 21:38:43.666777 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:43.666738 5932 generic.go:356] "Generic (PLEG): container finished" podID="1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" containerID="8025b3207801543c91b7cafafbfb0a614ef906ebbc4a7de88e6fdaf7e6228090" exitCode=0 Nov 03 21:38:43.667230 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:43.666830 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" event={"ID":"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8","Type":"ContainerDied","Data":"8025b3207801543c91b7cafafbfb0a614ef906ebbc4a7de88e6fdaf7e6228090"} Nov 03 21:38:44.160420 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:38:44.160372 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:38:44.853005 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:44.852973 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:44.921599 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:44.921557 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7\" (UniqueName: \"kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7\") pod \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " Nov 03 21:38:44.921837 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:44.921677 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca\") pod \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " Nov 03 21:38:44.922089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:44.922057 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca" pod "1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" (UID: "1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:38:44.923940 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:44.923909 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7" pod "1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" (UID: "1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8"). InnerVolumeSpecName "kube-api-access-qk5q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:38:45.040205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.039323 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca\") pod \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\" (UID: \"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8\") " Nov 03 21:38:45.040205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.039760 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:45.040205 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.039783 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk5q7\" (UniqueName: \"kubernetes.io/projected/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-kube-api-access-qk5q7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:45.058415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.058358 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca" pod "1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" (UID: "1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:38:45.144103 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.143667 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:38:45.700441 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.700405 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" Nov 03 21:38:45.700755 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.700420 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz" event={"ID":"1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8","Type":"ContainerDied","Data":"1a71c665219c27dfefa003572f89a99f29d3c748fe9735d5d7040af32d00fc57"} Nov 03 21:38:45.700973 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:38:45.700953 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a71c665219c27dfefa003572f89a99f29d3c748fe9735d5d7040af32d00fc57" Nov 03 21:39:01.588241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:01.588207 5932 scope.go:122] "RemoveContainer" containerID="cd2c35daaaf0782ca9e83bca47ed5bbeeccf5fee37149ff49a37b28d2ab7372a" Nov 03 21:39:12.180268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.180173 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z"] Nov 03 21:39:12.180985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.180674 5932 cpu_manager.go:414] "containerMap: removing container" podUID="1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" containerName="revoke-keys" Nov 03 21:39:12.180985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.180691 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" containerName="revoke-keys" Nov 03 21:39:12.180985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.180861 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" containerName="revoke-keys" Nov 03 21:39:12.185516 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.185494 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.281295 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.281258 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:39:12.281513 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.281328 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:39:12.281637 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.281600 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.281810 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.281787 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h2rn\" (UniqueName: \"kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.281881 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.281838 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.283837 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.283812 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z"] Nov 03 21:39:12.382933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.382891 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h2rn\" (UniqueName: \"kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.382933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.382947 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.383249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.383043 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.383840 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.383810 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.385461 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.385442 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.440204 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.440117 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h2rn\" (UniqueName: \"kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn\") pod \"maas-api-revoke-sub-3e31a506-tjn9z\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.498610 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.498567 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:12.872582 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:12.872480 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z"] Nov 03 21:39:12.878904 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:39:12.878870 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370dad93_7207_4390_89c1_b09eb5c81cd8.slice/crio-e59c19be2937ace9619a93fb5138d2030ded57f6f7c2c991fee4f84346863aa1 WatchSource:0}: Error finding container e59c19be2937ace9619a93fb5138d2030ded57f6f7c2c991fee4f84346863aa1: Status 404 returned error can't find the container with id e59c19be2937ace9619a93fb5138d2030ded57f6f7c2c991fee4f84346863aa1 Nov 03 21:39:13.867899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:13.867866 5932 generic.go:356] "Generic (PLEG): container finished" podID="370dad93-7207-4390-89c1-b09eb5c81cd8" containerID="d44f0e2170d2f631a095275317be0abb9426b6634a9a769aac1d1631e7620af5" exitCode=0 Nov 03 21:39:13.868385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:13.867918 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" event={"ID":"370dad93-7207-4390-89c1-b09eb5c81cd8","Type":"ContainerDied","Data":"d44f0e2170d2f631a095275317be0abb9426b6634a9a769aac1d1631e7620af5"} Nov 03 21:39:13.868385 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:13.867979 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" event={"ID":"370dad93-7207-4390-89c1-b09eb5c81cd8","Type":"ContainerStarted","Data":"e59c19be2937ace9619a93fb5138d2030ded57f6f7c2c991fee4f84346863aa1"} Nov 03 21:39:14.152311 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:39:14.152270 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:39:15.294329 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.294298 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:15.316351 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.316324 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn\" (UniqueName: \"kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn\") pod \"370dad93-7207-4390-89c1-b09eb5c81cd8\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " Nov 03 21:39:15.316535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.316382 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca\") pod \"370dad93-7207-4390-89c1-b09eb5c81cd8\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " Nov 03 21:39:15.316535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.316525 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca\") pod \"370dad93-7207-4390-89c1-b09eb5c81cd8\" (UID: \"370dad93-7207-4390-89c1-b09eb5c81cd8\") " Nov 03 21:39:15.316899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.316870 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca" pod "370dad93-7207-4390-89c1-b09eb5c81cd8" (UID: "370dad93-7207-4390-89c1-b09eb5c81cd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:39:15.318470 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.318439 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca" pod "370dad93-7207-4390-89c1-b09eb5c81cd8" (UID: "370dad93-7207-4390-89c1-b09eb5c81cd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:39:15.318560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.318526 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn" pod "370dad93-7207-4390-89c1-b09eb5c81cd8" (UID: "370dad93-7207-4390-89c1-b09eb5c81cd8"). InnerVolumeSpecName "kube-api-access-7h2rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:15.418119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.418088 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:15.418119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.418117 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/370dad93-7207-4390-89c1-b09eb5c81cd8-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:15.418119 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.418129 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h2rn\" (UniqueName: \"kubernetes.io/projected/370dad93-7207-4390-89c1-b09eb5c81cd8-kube-api-access-7h2rn\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:15.901700 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.901661 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" event={"ID":"370dad93-7207-4390-89c1-b09eb5c81cd8","Type":"ContainerDied","Data":"e59c19be2937ace9619a93fb5138d2030ded57f6f7c2c991fee4f84346863aa1"} Nov 03 21:39:15.901877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.901729 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59c19be2937ace9619a93fb5138d2030ded57f6f7c2c991fee4f84346863aa1" Nov 03 21:39:15.901877 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:15.901670 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z" Nov 03 21:39:33.065650 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:33.064861 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6"] Nov 03 21:39:33.103135 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:33.103094 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-r72d6"] Nov 03 21:39:34.075190 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:34.075139 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccbef9f-d4e6-4744-925c-5289a2d2db5d" path="/var/lib/kubelet/pods/cccbef9f-d4e6-4744-925c-5289a2d2db5d/volumes" Nov 03 21:39:41.745560 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.745510 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg"] Nov 03 21:39:41.746229 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.746206 5932 cpu_manager.go:414] "containerMap: removing container" podUID="370dad93-7207-4390-89c1-b09eb5c81cd8" containerName="revoke-keys" Nov 03 21:39:41.746331 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.746232 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="370dad93-7207-4390-89c1-b09eb5c81cd8" containerName="revoke-keys" Nov 03 21:39:41.746390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.746359 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="370dad93-7207-4390-89c1-b09eb5c81cd8" containerName="revoke-keys" Nov 03 21:39:41.749987 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.749963 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.774952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.774912 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkxkm\" (UniqueName: \"kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.774952 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.774956 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.775176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.775128 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.846725 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.846685 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 21:39:41.846907 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.846743 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cdqm5\"" Nov 03 21:39:41.849065 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.849041 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg"] Nov 03 21:39:41.876705 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.876671 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkxkm\" (UniqueName: \"kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.876897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.876709 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.876897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.876798 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.877502 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.877474 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.879256 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.879235 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:41.936015 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:41.935986 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkxkm\" (UniqueName: \"kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm\") pod \"maas-api-revoke-sub-141da78c-n4clg\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:42.063436 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:42.063360 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:42.442178 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:42.442144 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg"] Nov 03 21:39:42.449010 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:39:42.448980 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98191308_390e_40ee_8ec1_49762696e032.slice/crio-2cb68e9da6f854be785e185549f41889c6584dc3e2dbfbf49fb1ff630cfc4fd3 WatchSource:0}: Error finding container 2cb68e9da6f854be785e185549f41889c6584dc3e2dbfbf49fb1ff630cfc4fd3: Status 404 returned error can't find the container with id 2cb68e9da6f854be785e185549f41889c6584dc3e2dbfbf49fb1ff630cfc4fd3 Nov 03 21:39:43.130318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:43.130279 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" event={"ID":"98191308-390e-40ee-8ec1-49762696e032","Type":"ContainerStarted","Data":"302cb658d1ecec0f73970f0582f93805a4ee4ad84522ec3c035f41d7b71fd6b9"} Nov 03 21:39:43.130744 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:43.130341 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" event={"ID":"98191308-390e-40ee-8ec1-49762696e032","Type":"ContainerStarted","Data":"2cb68e9da6f854be785e185549f41889c6584dc3e2dbfbf49fb1ff630cfc4fd3"} Nov 03 21:39:43.237807 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:43.237739 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" podStartSLOduration=2.237720942 podStartE2EDuration="2.237720942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:43.237266002 +0000 UTC m=+4191.302507477" watchObservedRunningTime="2026-11-03 21:39:43.237720942 +0000 UTC m=+4191.302962418" Nov 03 21:39:44.138756 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:44.138718 5932 generic.go:356] "Generic (PLEG): container finished" podID="98191308-390e-40ee-8ec1-49762696e032" containerID="302cb658d1ecec0f73970f0582f93805a4ee4ad84522ec3c035f41d7b71fd6b9" exitCode=0 Nov 03 21:39:44.139200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:44.138827 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" event={"ID":"98191308-390e-40ee-8ec1-49762696e032","Type":"ContainerDied","Data":"302cb658d1ecec0f73970f0582f93805a4ee4ad84522ec3c035f41d7b71fd6b9"} Nov 03 21:39:44.151746 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:39:44.151713 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:39:45.291491 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.291459 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:45.415487 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.415400 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm\" (UniqueName: \"kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm\") pod \"98191308-390e-40ee-8ec1-49762696e032\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " Nov 03 21:39:45.415487 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.415479 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca\") pod \"98191308-390e-40ee-8ec1-49762696e032\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " Nov 03 21:39:45.415775 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.415571 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca\") pod \"98191308-390e-40ee-8ec1-49762696e032\" (UID: \"98191308-390e-40ee-8ec1-49762696e032\") " Nov 03 21:39:45.415937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.415907 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca" pod "98191308-390e-40ee-8ec1-49762696e032" (UID: "98191308-390e-40ee-8ec1-49762696e032"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:39:45.417663 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.417614 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca" pod "98191308-390e-40ee-8ec1-49762696e032" (UID: "98191308-390e-40ee-8ec1-49762696e032"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:39:45.417770 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.417681 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm" pod "98191308-390e-40ee-8ec1-49762696e032" (UID: "98191308-390e-40ee-8ec1-49762696e032"). InnerVolumeSpecName "kube-api-access-xkxkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:39:45.516481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.516440 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkxkm\" (UniqueName: \"kubernetes.io/projected/98191308-390e-40ee-8ec1-49762696e032-kube-api-access-xkxkm\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:45.516481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.516471 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98191308-390e-40ee-8ec1-49762696e032-maas-api-openshift-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:45.516481 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:45.516487 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98191308-390e-40ee-8ec1-49762696e032-maas-api-xks-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:39:46.151698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:46.151656 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" event={"ID":"98191308-390e-40ee-8ec1-49762696e032","Type":"ContainerDied","Data":"2cb68e9da6f854be785e185549f41889c6584dc3e2dbfbf49fb1ff630cfc4fd3"} Nov 03 21:39:46.151885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:46.151718 5932 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cb68e9da6f854be785e185549f41889c6584dc3e2dbfbf49fb1ff630cfc4fd3" Nov 03 21:39:46.151885 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:46.151673 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg" Nov 03 21:39:54.969811 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:54.969778 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-zf8ld"] Nov 03 21:39:54.970268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:54.970249 5932 cpu_manager.go:414] "containerMap: removing container" podUID="98191308-390e-40ee-8ec1-49762696e032" containerName="revoke-keys" Nov 03 21:39:54.970323 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:54.970270 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98191308-390e-40ee-8ec1-49762696e032" containerName="revoke-keys" Nov 03 21:39:54.970382 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:54.970372 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="98191308-390e-40ee-8ec1-49762696e032" containerName="revoke-keys" Nov 03 21:39:54.973847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:54.973826 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.069186 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.069146 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r8zvx\"" Nov 03 21:39:55.069871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.069490 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 21:39:55.069871 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.069523 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 21:39:55.073388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.073362 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-zf8ld"] Nov 03 21:39:55.111251 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.111217 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.111440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.111260 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.111440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.111405 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bgc\" (UniqueName: \"kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.111539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.111481 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.212020 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.211984 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bgc\" (UniqueName: \"kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.212240 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.212047 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.212240 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.212081 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.212240 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.212106 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.212790 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.212769 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.214761 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.214731 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.214872 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.214780 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.269393 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.269314 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bgc\" (UniqueName: \"kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc\") pod \"maas-api-844bc99d-zf8ld\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.289210 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.289170 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:55.662263 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.662230 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-zf8ld"] Nov 03 21:39:55.668797 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:39:55.668766 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3926617a_92c7_4479_b881_a31a10eb0b51.slice/crio-1da63cca0727c23f3efdc64ea5985a17dad4449a46f2cf6dd6cf241f84a41ecc WatchSource:0}: Error finding container 1da63cca0727c23f3efdc64ea5985a17dad4449a46f2cf6dd6cf241f84a41ecc: Status 404 returned error can't find the container with id 1da63cca0727c23f3efdc64ea5985a17dad4449a46f2cf6dd6cf241f84a41ecc Nov 03 21:39:55.670174 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:55.670157 5932 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 21:39:56.220284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:56.220249 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" event={"ID":"3926617a-92c7-4479-b881-a31a10eb0b51","Type":"ContainerStarted","Data":"1da63cca0727c23f3efdc64ea5985a17dad4449a46f2cf6dd6cf241f84a41ecc"} Nov 03 21:39:56.372035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:56.372007 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:39:56.389434 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:56.389408 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:39:57.228098 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:57.228063 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" event={"ID":"3926617a-92c7-4479-b881-a31a10eb0b51","Type":"ContainerStarted","Data":"6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0"} Nov 03 21:39:57.228529 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:57.228194 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:39:57.340009 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:39:57.339952 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" podStartSLOduration=2.422990321 podStartE2EDuration="3.339934554s" totalImagesPullingTime="916.944233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:39:57.338389513 +0000 UTC m=+4205.403630987" watchObservedRunningTime="2026-11-03 21:39:57.339934554 +0000 UTC m=+4205.405176028" Nov 03 21:40:01.048426 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:01.048389 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22"] Nov 03 21:40:01.094550 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:01.094511 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-fnf22"] Nov 03 21:40:01.988338 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:01.988298 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c"] Nov 03 21:40:01.992232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:01.992212 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.041299 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.041260 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4f1680df-openshift-default-dockercfg-bv48z\"" Nov 03 21:40:02.054177 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.054146 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4537cb2c-e53b-41f7-b53f-9d43eaf52233" path="/var/lib/kubelet/pods/4537cb2c-e53b-41f7-b53f-9d43eaf52233/volumes" Nov 03 21:40:02.080988 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.080944 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.080992 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081099 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081156 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081147 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksf6\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081218 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081238 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081272 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081334 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081289 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.081473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.081396 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.090665 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.090614 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c"] Nov 03 21:40:02.182794 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.182757 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksf6\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.182995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.182818 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.182995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.182847 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.182995 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.182891 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183029 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183193 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183124 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183203 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183239 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183315 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183288 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183313 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183454 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183367 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183555 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183537 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183672 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183646 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.183776 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.183760 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.186012 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.185978 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.186188 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.186169 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.260828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.260695 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksf6\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.264318 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.264289 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token\") pod \"e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.305237 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.305203 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:02.731927 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.731890 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c"] Nov 03 21:40:02.737898 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:40:02.737866 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ec9220_558b_4b2f_b09b_d7127d8b8956.slice/crio-5da10c455838cf23a7e4e0d6c288bccfbb34e34d2178a42c999acc50061e3add WatchSource:0}: Error finding container 5da10c455838cf23a7e4e0d6c288bccfbb34e34d2178a42c999acc50061e3add: Status 404 returned error can't find the container with id 5da10c455838cf23a7e4e0d6c288bccfbb34e34d2178a42c999acc50061e3add Nov 03 21:40:02.740241 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.740206 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:40:02.740319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.740274 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:40:02.740319 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:02.740303 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:40:03.240400 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.240369 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:40:03.269510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.269477 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" event={"ID":"51ec9220-558b-4b2f-b09b-d7127d8b8956","Type":"ContainerStarted","Data":"ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e"} Nov 03 21:40:03.269706 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.269528 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" event={"ID":"51ec9220-558b-4b2f-b09b-d7127d8b8956","Type":"ContainerStarted","Data":"5da10c455838cf23a7e4e0d6c288bccfbb34e34d2178a42c999acc50061e3add"} Nov 03 21:40:03.305402 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.305365 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:03.310825 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.310797 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:03.465284 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.465225 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podStartSLOduration=2.465205489 podStartE2EDuration="2.465205489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:03.464730624 +0000 UTC m=+4211.529972099" watchObservedRunningTime="2026-11-03 21:40:03.465205489 +0000 UTC m=+4211.530446966" Nov 03 21:40:03.693487 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.693446 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g"] Nov 03 21:40:03.697320 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.697295 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.752475 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.752437 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g"] Nov 03 21:40:03.800901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.800861 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrq2\" (UniqueName: \"kubernetes.io/projected/c27eed1d-4d36-4d6b-a264-9217a193a291-kube-api-access-mlrq2\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.801108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.800943 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27eed1d-4d36-4d6b-a264-9217a193a291-openshift-service-ca\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.801108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.801017 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27eed1d-4d36-4d6b-a264-9217a193a291-metrics-certs\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.801108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.801083 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c27eed1d-4d36-4d6b-a264-9217a193a291-maas-api-tls\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.902022 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.901988 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrq2\" (UniqueName: \"kubernetes.io/projected/c27eed1d-4d36-4d6b-a264-9217a193a291-kube-api-access-mlrq2\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.902370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.902061 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27eed1d-4d36-4d6b-a264-9217a193a291-openshift-service-ca\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.902370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.902093 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27eed1d-4d36-4d6b-a264-9217a193a291-metrics-certs\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.902370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.902134 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c27eed1d-4d36-4d6b-a264-9217a193a291-maas-api-tls\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.902902 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.902877 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27eed1d-4d36-4d6b-a264-9217a193a291-openshift-service-ca\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.904653 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.904616 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c27eed1d-4d36-4d6b-a264-9217a193a291-maas-api-tls\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.904772 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.904752 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c27eed1d-4d36-4d6b-a264-9217a193a291-metrics-certs\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:03.959420 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:03.959351 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrq2\" (UniqueName: \"kubernetes.io/projected/c27eed1d-4d36-4d6b-a264-9217a193a291-kube-api-access-mlrq2\") pod \"maas-api-bbfc75f54-szn2g\" (UID: \"c27eed1d-4d36-4d6b-a264-9217a193a291\") " pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:04.010938 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:04.010903 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:04.276866 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:04.276789 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:04.278468 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:04.278442 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:04.391220 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:04.391181 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g"] Nov 03 21:40:04.398879 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:40:04.398846 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27eed1d_4d36_4d6b_a264_9217a193a291.slice/crio-e6fc2d048d1d2c7528c42d3474849f6e293f6413d54c31c5726b46e111b417d0 WatchSource:0}: Error finding container e6fc2d048d1d2c7528c42d3474849f6e293f6413d54c31c5726b46e111b417d0: Status 404 returned error can't find the container with id e6fc2d048d1d2c7528c42d3474849f6e293f6413d54c31c5726b46e111b417d0 Nov 03 21:40:05.284243 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:05.284202 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" event={"ID":"c27eed1d-4d36-4d6b-a264-9217a193a291","Type":"ContainerStarted","Data":"e6fc2d048d1d2c7528c42d3474849f6e293f6413d54c31c5726b46e111b417d0"} Nov 03 21:40:06.293390 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:06.293349 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" event={"ID":"c27eed1d-4d36-4d6b-a264-9217a193a291","Type":"ContainerStarted","Data":"59626ce52d88dfa15c2598e394438e8fadc9528755f190d52b8d801efd3bfaef"} Nov 03 21:40:06.293908 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:06.293467 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:06.405314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:06.405253 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" podStartSLOduration=2.460688453 podStartE2EDuration="3.405232932s" totalImagesPullingTime="944.544479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:40:06.403908145 +0000 UTC m=+4214.469149645" watchObservedRunningTime="2026-11-03 21:40:06.405232932 +0000 UTC m=+4214.470474409" Nov 03 21:40:12.304782 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.304750 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bbfc75f54-szn2g" Nov 03 21:40:12.465994 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.465958 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-zf8ld"] Nov 03 21:40:12.466252 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.466222 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" podUID="3926617a-92c7-4479-b881-a31a10eb0b51" containerName="maas-api" containerID="cri-o://6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0" gracePeriod=30 Nov 03 21:40:12.492398 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:40:12.492370 5932 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3926617a_92c7_4479_b881_a31a10eb0b51.slice/crio-6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 21:40:12.732415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.732389 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:40:12.802662 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.802616 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc\" (UniqueName: \"kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc\") pod \"3926617a-92c7-4479-b881-a31a10eb0b51\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " Nov 03 21:40:12.802913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.802708 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca\") pod \"3926617a-92c7-4479-b881-a31a10eb0b51\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " Nov 03 21:40:12.802913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.802737 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs\") pod \"3926617a-92c7-4479-b881-a31a10eb0b51\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " Nov 03 21:40:12.802913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.802806 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls\") pod \"3926617a-92c7-4479-b881-a31a10eb0b51\" (UID: \"3926617a-92c7-4479-b881-a31a10eb0b51\") " Nov 03 21:40:12.803261 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.803227 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca" pod "3926617a-92c7-4479-b881-a31a10eb0b51" (UID: "3926617a-92c7-4479-b881-a31a10eb0b51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:40:12.805133 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.805101 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs" pod "3926617a-92c7-4479-b881-a31a10eb0b51" (UID: "3926617a-92c7-4479-b881-a31a10eb0b51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:12.805288 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.805111 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls" pod "3926617a-92c7-4479-b881-a31a10eb0b51" (UID: "3926617a-92c7-4479-b881-a31a10eb0b51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:40:12.805355 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.805302 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc" pod "3926617a-92c7-4479-b881-a31a10eb0b51" (UID: "3926617a-92c7-4479-b881-a31a10eb0b51"). InnerVolumeSpecName "kube-api-access-g8bgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:40:12.904293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.904248 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bgc\" (UniqueName: \"kubernetes.io/projected/3926617a-92c7-4479-b881-a31a10eb0b51-kube-api-access-g8bgc\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:12.904293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.904284 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3926617a-92c7-4479-b881-a31a10eb0b51-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:12.904293 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.904298 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:12.904562 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:12.904310 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3926617a-92c7-4479-b881-a31a10eb0b51-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:40:13.336388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.336292 5932 generic.go:356] "Generic (PLEG): container finished" podID="3926617a-92c7-4479-b881-a31a10eb0b51" containerID="6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0" exitCode=0 Nov 03 21:40:13.336388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.336342 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" Nov 03 21:40:13.336388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.336355 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" event={"ID":"3926617a-92c7-4479-b881-a31a10eb0b51","Type":"ContainerDied","Data":"6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0"} Nov 03 21:40:13.336977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.336406 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-844bc99d-zf8ld" event={"ID":"3926617a-92c7-4479-b881-a31a10eb0b51","Type":"ContainerDied","Data":"1da63cca0727c23f3efdc64ea5985a17dad4449a46f2cf6dd6cf241f84a41ecc"} Nov 03 21:40:13.336977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.336438 5932 scope.go:122] "RemoveContainer" containerID="6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0" Nov 03 21:40:13.358489 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.358461 5932 scope.go:122] "RemoveContainer" containerID="6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0" Nov 03 21:40:13.359097 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:40:13.359067 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0\": container with ID starting with 6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0 not found: ID does not exist" containerID="6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0" Nov 03 21:40:13.359243 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.359105 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0"} err="failed to get container status \"6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0\": rpc error: code = NotFound desc = could not find container \"6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0\": container with ID starting with 6ccfa65a41b33a871bd4a42d5480041fe1ed857062003ea1563d16bb84ae21c0 not found: ID does not exist" Nov 03 21:40:13.489824 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.489778 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-zf8ld"] Nov 03 21:40:13.536070 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:13.536030 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-844bc99d-zf8ld"] Nov 03 21:40:14.051313 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:14.051278 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3926617a-92c7-4479-b881-a31a10eb0b51" path="/var/lib/kubelet/pods/3926617a-92c7-4479-b881-a31a10eb0b51/volumes" Nov 03 21:40:14.151081 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:40:14.151045 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:40:29.482500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:29.482466 5932 generic.go:356] "Generic (PLEG): container finished" podID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerID="ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e" exitCode=0 Nov 03 21:40:29.483015 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:29.482538 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" event={"ID":"51ec9220-558b-4b2f-b09b-d7127d8b8956","Type":"ContainerDied","Data":"ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e"} Nov 03 21:40:29.483015 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:29.482974 5932 scope.go:122] "RemoveContainer" containerID="ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e" Nov 03 21:40:29.484244 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:29.484218 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:40:29.484303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:29.484286 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:40:29.484350 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:29.484317 5932 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 21:40:30.490593 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:30.490546 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" event={"ID":"51ec9220-558b-4b2f-b09b-d7127d8b8956","Type":"ContainerStarted","Data":"290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e"} Nov 03 21:40:31.305739 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:31.305698 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:31.307277 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:31.307242 5932 patch_prober.go:29] interesting pod/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 21:40:31.307439 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:31.307311 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 21:40:32.306430 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:32.306394 5932 patch_prober.go:29] interesting pod/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 21:40:32.307091 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:32.306450 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 21:40:33.306521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:33.306428 5932 patch_prober.go:29] interesting pod/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 21:40:33.306521 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:33.306484 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 21:40:34.276937 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:34.276899 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:34.306476 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:34.306435 5932 patch_prober.go:29] interesting pod/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 21:40:34.306725 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:34.306499 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 21:40:35.305648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:35.305584 5932 patch_prober.go:29] interesting pod/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 21:40:35.305930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:35.305667 5932 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 21:40:36.311096 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:36.311058 5932 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:36.312471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:40:36.312438 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:40:44.151381 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:40:44.151337 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:41:01.621144 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:01.621104 5932 scope.go:122] "RemoveContainer" containerID="8c29a73c769fadf7022f632eed3ce10b03a05a1f53f798b3f3ef4f3d059984f1" Nov 03 21:41:01.630977 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:01.630951 5932 scope.go:122] "RemoveContainer" containerID="eb6e436e240814643c0b16da256975402636a99695f71069cef1fa48c8d4a2c0" Nov 03 21:41:14.151421 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:41:14.151384 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:41:18.348878 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:18.348828 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c"] Nov 03 21:41:18.349310 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:18.349099 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" containerID="cri-o://290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e" gracePeriod=30 Nov 03 21:41:21.313573 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:21.313536 5932 patch_prober.go:29] interesting pod/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 21:41:21.314108 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:21.313596 5932 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 21:41:23.609016 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.608991 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:41:23.735340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735309 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735364 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735385 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735433 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735454 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735503 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735552 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735598 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:23.735869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735615 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735672 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert\") pod \"51ec9220-558b-4b2f-b09b-d7127d8b8956\" (UID: \"51ec9220-558b-4b2f-b09b-d7127d8b8956\") " Nov 03 21:41:23.735869 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735797 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:23.736143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.735938 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:23.736143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.736088 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:41:23.736143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.736107 5932 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-credential-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.736143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.736128 5932 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-data\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.736143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.736143 5932 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-socket\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.736415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.736290 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:23.737843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.737819 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:23.737968 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.737831 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 21:41:23.738080 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.738065 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "kube-api-access-lksf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:41:23.738143 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.738112 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo" pod "51ec9220-558b-4b2f-b09b-d7127d8b8956" (UID: "51ec9220-558b-4b2f-b09b-d7127d8b8956"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 21:41:23.803375 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.803342 5932 generic.go:356] "Generic (PLEG): container finished" podID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerID="290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e" exitCode=0 Nov 03 21:41:23.803532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.803398 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" Nov 03 21:41:23.803532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.803424 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" event={"ID":"51ec9220-558b-4b2f-b09b-d7127d8b8956","Type":"ContainerDied","Data":"290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e"} Nov 03 21:41:23.803532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.803478 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c" event={"ID":"51ec9220-558b-4b2f-b09b-d7127d8b8956","Type":"ContainerDied","Data":"5da10c455838cf23a7e4e0d6c288bccfbb34e34d2178a42c999acc50061e3add"} Nov 03 21:41:23.803532 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.803501 5932 scope.go:122] "RemoveContainer" containerID="290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e" Nov 03 21:41:23.814272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.814252 5932 scope.go:122] "RemoveContainer" containerID="ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e" Nov 03 21:41:23.824997 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.824976 5932 scope.go:122] "RemoveContainer" containerID="290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e" Nov 03 21:41:23.825255 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:41:23.825230 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e\": container with ID starting with 290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e not found: ID does not exist" containerID="290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e" Nov 03 21:41:23.825316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.825268 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e"} err="failed to get container status \"290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e\": rpc error: code = NotFound desc = could not find container \"290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e\": container with ID starting with 290b17ab882ab57da581874970b81a0a161ad04686f1db7d7e54f2f28749458e not found: ID does not exist" Nov 03 21:41:23.825316 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.825295 5932 scope.go:122] "RemoveContainer" containerID="ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e" Nov 03 21:41:23.825570 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:41:23.825551 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e\": container with ID starting with ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e not found: ID does not exist" containerID="ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e" Nov 03 21:41:23.825643 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.825576 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e"} err="failed to get container status \"ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e\": rpc error: code = NotFound desc = could not find container \"ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e\": container with ID starting with ab0efff10fc448f78b5d1a18c1eb9e46252da92083f7b28e802d567476c48d9e not found: ID does not exist" Nov 03 21:41:23.836985 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.836966 5932 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-workload-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.837081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.836986 5932 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51ec9220-558b-4b2f-b09b-d7127d8b8956-istiod-ca-cert\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.837081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.836996 5932 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-podinfo\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.837081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.837007 5932 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-token\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.837081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.837016 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lksf6\" (UniqueName: \"kubernetes.io/projected/51ec9220-558b-4b2f-b09b-d7127d8b8956-kube-api-access-lksf6\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.837081 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.837026 5932 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51ec9220-558b-4b2f-b09b-d7127d8b8956-istio-envoy\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:41:23.958918 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.958894 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c"] Nov 03 21:41:23.962781 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:23.962763 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4f1680df-openshift-default-65d9bf4d87-dlv2c"] Nov 03 21:41:24.050432 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:24.050364 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" path="/var/lib/kubelet/pods/51ec9220-558b-4b2f-b09b-d7127d8b8956/volumes" Nov 03 21:41:36.720536 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720503 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc"] Nov 03 21:41:36.720956 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720942 5932 cpu_manager.go:414] "containerMap: removing container" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" Nov 03 21:41:36.721002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720957 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" Nov 03 21:41:36.721002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720967 5932 cpu_manager.go:414] "containerMap: removing container" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" Nov 03 21:41:36.721002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720972 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" Nov 03 21:41:36.721002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720983 5932 cpu_manager.go:414] "containerMap: removing container" podUID="3926617a-92c7-4479-b881-a31a10eb0b51" containerName="maas-api" Nov 03 21:41:36.721002 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.720988 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3926617a-92c7-4479-b881-a31a10eb0b51" containerName="maas-api" Nov 03 21:41:36.721169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.721090 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" Nov 03 21:41:36.721169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.721096 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="51ec9220-558b-4b2f-b09b-d7127d8b8956" containerName="istio-proxy" Nov 03 21:41:36.721169 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.721102 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="3926617a-92c7-4479-b881-a31a10eb0b51" containerName="maas-api" Nov 03 21:41:36.724440 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.724422 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:36.775901 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.775870 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-bbda3a\"" Nov 03 21:41:36.821725 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.821698 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-bbda3a-dockercfg-xp8cx\"" Nov 03 21:41:36.824044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.824022 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc"] Nov 03 21:41:36.876975 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.876946 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvn7\" (UniqueName: \"kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7\") pod \"payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:36.877120 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.876979 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:36.978066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.977980 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvn7\" (UniqueName: \"kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7\") pod \"payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:36.978066 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.978026 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:36.978798 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:36.978777 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:37.035500 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.035461 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvn7\" (UniqueName: \"kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7\") pod \"payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:37.037321 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.037287 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:37.409738 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.409702 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc"] Nov 03 21:41:37.415704 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:41:37.415674 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb72260_cb01_412f_9cc5_312d97ecba96.slice/crio-34a67e78e74a2adc0e460898f6adef0274c84034684a9f28bdd2e27120d6c095 WatchSource:0}: Error finding container 34a67e78e74a2adc0e460898f6adef0274c84034684a9f28bdd2e27120d6c095: Status 404 returned error can't find the container with id 34a67e78e74a2adc0e460898f6adef0274c84034684a9f28bdd2e27120d6c095 Nov 03 21:41:37.885847 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.885810 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" event={"ID":"9bb72260-cb01-412f-9cc5-312d97ecba96","Type":"ContainerStarted","Data":"55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560"} Nov 03 21:41:37.886249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.885873 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" event={"ID":"9bb72260-cb01-412f-9cc5-312d97ecba96","Type":"ContainerStarted","Data":"34a67e78e74a2adc0e460898f6adef0274c84034684a9f28bdd2e27120d6c095"} Nov 03 21:41:37.886249 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.885905 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:41:37.996281 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:37.996228 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" podStartSLOduration=1.9962136190000002 podStartE2EDuration="1.996213619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:37.995171453 +0000 UTC m=+4306.060412918" watchObservedRunningTime="2026-11-03 21:41:37.996213619 +0000 UTC m=+4306.061455110" Nov 03 21:41:39.118302 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.118266 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7"] Nov 03 21:41:39.122124 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.122104 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.170671 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.170615 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-bbda3a-serving-cert\"" Nov 03 21:41:39.219200 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.219160 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7"] Nov 03 21:41:39.301354 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.301320 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.301539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.301381 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xk2\" (UniqueName: \"kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.301539 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.301467 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.301616 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.301597 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.402577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.402535 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xk2\" (UniqueName: \"kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.402577 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.402579 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.402835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.402670 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.402835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.402745 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.403471 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.403444 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.405362 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.405332 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.405460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.405375 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.461473 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.461436 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xk2\" (UniqueName: \"kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2\") pod \"maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:39.736138 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:39.736048 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:40.110564 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:40.110483 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7"] Nov 03 21:41:40.116689 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:41:40.116658 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b384176_2244_4a40_b7b6_c88e6e99fad6.slice/crio-7d26752b80970e008c0025cd800e47394917f397da595add0eb456f8ef6bd6d7 WatchSource:0}: Error finding container 7d26752b80970e008c0025cd800e47394917f397da595add0eb456f8ef6bd6d7: Status 404 returned error can't find the container with id 7d26752b80970e008c0025cd800e47394917f397da595add0eb456f8ef6bd6d7 Nov 03 21:41:40.904592 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:40.904555 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" event={"ID":"6b384176-2244-4a40-b7b6-c88e6e99fad6","Type":"ContainerStarted","Data":"7d26752b80970e008c0025cd800e47394917f397da595add0eb456f8ef6bd6d7"} Nov 03 21:41:41.911759 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:41.911714 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" event={"ID":"6b384176-2244-4a40-b7b6-c88e6e99fad6","Type":"ContainerStarted","Data":"520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c"} Nov 03 21:41:41.912265 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:41.911845 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:42.023828 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:42.023767 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" podStartSLOduration=2.152000659 podStartE2EDuration="3.023753333s" totalImagesPullingTime="871.752674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 21:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:41:42.021912058 +0000 UTC m=+4310.087153533" watchObservedRunningTime="2026-11-03 21:41:42.023753333 +0000 UTC m=+4310.088994806" Nov 03 21:41:44.150419 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:41:44.150375 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:41:47.923508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:47.923477 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:41:48.894640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:41:48.894597 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:42:14.150077 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:42:14.150001 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:42:44.150867 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:42:44.150827 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:43:14.149717 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:43:14.149680 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:43:29.555447 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:29.555411 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc"] Nov 03 21:43:34.556147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.556045 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" podUID="9bb72260-cb01-412f-9cc5-312d97ecba96" containerName="payload-processing" containerID="cri-o://55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560" gracePeriod=25 Nov 03 21:43:34.812652 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.812572 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:43:34.832369 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.832336 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7\" (UniqueName: \"kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7\") pod \"9bb72260-cb01-412f-9cc5-312d97ecba96\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " Nov 03 21:43:34.832537 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.832483 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume\") pod \"9bb72260-cb01-412f-9cc5-312d97ecba96\" (UID: \"9bb72260-cb01-412f-9cc5-312d97ecba96\") " Nov 03 21:43:34.832843 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.832820 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume" pod "9bb72260-cb01-412f-9cc5-312d97ecba96" (UID: "9bb72260-cb01-412f-9cc5-312d97ecba96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:34.834410 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.834389 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7" pod "9bb72260-cb01-412f-9cc5-312d97ecba96" (UID: "9bb72260-cb01-412f-9cc5-312d97ecba96"). InnerVolumeSpecName "kube-api-access-xsvn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:34.933528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.933482 5932 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bb72260-cb01-412f-9cc5-312d97ecba96-plugins-config-volume\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:34.933528 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:34.933522 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsvn7\" (UniqueName: \"kubernetes.io/projected/9bb72260-cb01-412f-9cc5-312d97ecba96-kube-api-access-xsvn7\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:35.558523 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.558486 5932 generic.go:356] "Generic (PLEG): container finished" podID="9bb72260-cb01-412f-9cc5-312d97ecba96" containerID="55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560" exitCode=0 Nov 03 21:43:35.559057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.558554 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" Nov 03 21:43:35.559057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.558576 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" event={"ID":"9bb72260-cb01-412f-9cc5-312d97ecba96","Type":"ContainerDied","Data":"55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560"} Nov 03 21:43:35.559057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.558682 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc" event={"ID":"9bb72260-cb01-412f-9cc5-312d97ecba96","Type":"ContainerDied","Data":"34a67e78e74a2adc0e460898f6adef0274c84034684a9f28bdd2e27120d6c095"} Nov 03 21:43:35.559057 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.558708 5932 scope.go:122] "RemoveContainer" containerID="55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560" Nov 03 21:43:35.571268 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.571243 5932 scope.go:122] "RemoveContainer" containerID="55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560" Nov 03 21:43:35.571673 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:43:35.571652 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560\": container with ID starting with 55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560 not found: ID does not exist" containerID="55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560" Nov 03 21:43:35.571768 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.571681 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560"} err="failed to get container status \"55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560\": rpc error: code = NotFound desc = could not find container \"55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560\": container with ID starting with 55a0557fe178eb662ed5ffdbad2d0f334c7e9bd8f5084f44908bc6fd36738560 not found: ID does not exist" Nov 03 21:43:35.712698 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.712659 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc"] Nov 03 21:43:35.716899 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:35.716872 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-bbda3a-66b4847f7-4b6vc"] Nov 03 21:43:36.050340 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:36.050307 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb72260-cb01-412f-9cc5-312d97ecba96" path="/var/lib/kubelet/pods/9bb72260-cb01-412f-9cc5-312d97ecba96/volumes" Nov 03 21:43:39.277728 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.277691 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7"] Nov 03 21:43:39.278176 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.277899 5932 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" podUID="6b384176-2244-4a40-b7b6-c88e6e99fad6" containerName="maas-api" containerID="cri-o://520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c" gracePeriod=30 Nov 03 21:43:39.549973 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.549571 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:43:39.579232 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.579200 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca\") pod \"6b384176-2244-4a40-b7b6-c88e6e99fad6\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " Nov 03 21:43:39.579413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.579249 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2\" (UniqueName: \"kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2\") pod \"6b384176-2244-4a40-b7b6-c88e6e99fad6\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " Nov 03 21:43:39.579413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.579332 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs\") pod \"6b384176-2244-4a40-b7b6-c88e6e99fad6\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " Nov 03 21:43:39.579413 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.579383 5932 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls\") pod \"6b384176-2244-4a40-b7b6-c88e6e99fad6\" (UID: \"6b384176-2244-4a40-b7b6-c88e6e99fad6\") " Nov 03 21:43:39.579647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.579596 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca" pod "6b384176-2244-4a40-b7b6-c88e6e99fad6" (UID: "6b384176-2244-4a40-b7b6-c88e6e99fad6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 21:43:39.579819 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.579797 5932 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b384176-2244-4a40-b7b6-c88e6e99fad6-openshift-service-ca\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:39.581545 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.581512 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2" pod "6b384176-2244-4a40-b7b6-c88e6e99fad6" (UID: "6b384176-2244-4a40-b7b6-c88e6e99fad6"). InnerVolumeSpecName "kube-api-access-c5xk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 21:43:39.581730 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.581706 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs" pod "6b384176-2244-4a40-b7b6-c88e6e99fad6" (UID: "6b384176-2244-4a40-b7b6-c88e6e99fad6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:39.581822 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.581803 5932 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls" pod "6b384176-2244-4a40-b7b6-c88e6e99fad6" (UID: "6b384176-2244-4a40-b7b6-c88e6e99fad6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 21:43:39.584010 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.583985 5932 generic.go:356] "Generic (PLEG): container finished" podID="6b384176-2244-4a40-b7b6-c88e6e99fad6" containerID="520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c" exitCode=0 Nov 03 21:43:39.584681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.584033 5932 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" Nov 03 21:43:39.584681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.584069 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" event={"ID":"6b384176-2244-4a40-b7b6-c88e6e99fad6","Type":"ContainerDied","Data":"520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c"} Nov 03 21:43:39.584681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.584126 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7" event={"ID":"6b384176-2244-4a40-b7b6-c88e6e99fad6","Type":"ContainerDied","Data":"7d26752b80970e008c0025cd800e47394917f397da595add0eb456f8ef6bd6d7"} Nov 03 21:43:39.584681 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.584150 5932 scope.go:122] "RemoveContainer" containerID="520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c" Nov 03 21:43:39.603872 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.603855 5932 scope.go:122] "RemoveContainer" containerID="520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c" Nov 03 21:43:39.604175 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:43:39.604158 5932 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c\": container with ID starting with 520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c not found: ID does not exist" containerID="520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c" Nov 03 21:43:39.604221 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.604183 5932 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c"} err="failed to get container status \"520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c\": rpc error: code = NotFound desc = could not find container \"520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c\": container with ID starting with 520dc2286eeaaa1208248d1290abc9170fc640c14613fe566e295e5b8262e40c not found: ID does not exist" Nov 03 21:43:39.680930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.680879 5932 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5xk2\" (UniqueName: \"kubernetes.io/projected/6b384176-2244-4a40-b7b6-c88e6e99fad6-kube-api-access-c5xk2\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:39.680930 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.680926 5932 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-metrics-certs\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:39.681129 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.680941 5932 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b384176-2244-4a40-b7b6-c88e6e99fad6-maas-api-tls\") on node \"ip-10-0-138-52.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 21:43:39.765425 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.765390 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7"] Nov 03 21:43:39.770664 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:39.770639 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-bbda3a-76477c5559-qfqn7"] Nov 03 21:43:40.051766 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:40.051730 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b384176-2244-4a40-b7b6-c88e6e99fad6" path="/var/lib/kubelet/pods/6b384176-2244-4a40-b7b6-c88e6e99fad6/volumes" Nov 03 21:43:44.150053 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:43:44.150008 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:43:45.044595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:45.044561 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz"] Nov 03 21:43:45.090581 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:45.090547 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-lswwz"] Nov 03 21:43:46.050347 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:43:46.050312 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8" path="/var/lib/kubelet/pods/1b8dcafa-92a8-4bcd-9ffd-01b2e69513c8/volumes" Nov 03 21:44:14.150327 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:44:14.150291 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:44:16.050603 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:16.050565 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z"] Nov 03 21:44:16.091188 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:16.091158 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-tjn9z"] Nov 03 21:44:18.051222 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:18.051190 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="370dad93-7207-4390-89c1-b09eb5c81cd8" path="/var/lib/kubelet/pods/370dad93-7207-4390-89c1-b09eb5c81cd8/volumes" Nov 03 21:44:44.149172 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:44:44.149136 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:44:46.052352 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:46.052319 5932 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg"] Nov 03 21:44:46.052750 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:46.052367 5932 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-141da78c-n4clg"] Nov 03 21:44:48.050269 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:48.050237 5932 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98191308-390e-40ee-8ec1-49762696e032" path="/var/lib/kubelet/pods/98191308-390e-40ee-8ec1-49762696e032/volumes" Nov 03 21:44:56.416692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:56.416658 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:44:56.431855 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:44:56.431833 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:45:01.720648 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:01.719845 5932 scope.go:122] "RemoveContainer" containerID="8025b3207801543c91b7cafafbfb0a614ef906ebbc4a7de88e6fdaf7e6228090" Nov 03 21:45:14.149167 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:45:14.149087 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:45:22.692933 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:22.692899 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9nk67_5bd2c162-bfec-4ee1-8b7c-ee3418b07532/manager/0.log" Nov 03 21:45:23.566595 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:23.566554 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcb8dd554-dczcz_4166cdea-dbd8-4a99-b275-ece179b45b9a/manager/0.log" Nov 03 21:45:24.482415 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:24.482364 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kqjhf_628e6e07-734c-4a91-81e4-21c61545b56b/proxy/0.log" Nov 03 21:45:25.533789 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:25.533760 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bbfc75f54-szn2g_c27eed1d-4d36-4d6b-a264-9217a193a291/maas-api/0.log" Nov 03 21:45:26.499551 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:26.499518 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2h456_084ba8dc-5495-4217-86b3-5b253ce4d3cc/postgres/0.log" Nov 03 21:45:28.491424 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:28.491390 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdz7m_b3bde11f-a135-4286-9b8e-c9c4c1bf3863/manager/0.log" Nov 03 21:45:28.810386 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:28.810301 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-j4td6_141d9b7c-0036-47c1-8299-b0f83f14026e/manager/0.log" Nov 03 21:45:29.923676 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:29.923646 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7vr97_a4eb75db-6377-44f0-8c02-9ed350b94153/manager/0.log" Nov 03 21:45:30.791339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:30.791298 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg_f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e/istio-proxy/0.log" Nov 03 21:45:31.005282 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:31.005235 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4zt6g_f5b16b74-0aeb-4ff9-b150-d9f8b3693d02/discovery/0.log" Nov 03 21:45:31.697929 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:31.697896 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-45kq6_82930fab-d2aa-49e9-b83e-bcbc34c20ba5/istio-proxy/0.log" Nov 03 21:45:32.187147 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:32.187115 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw_37560725-01f3-4f64-a350-31463f2bb770/istio-proxy/0.log" Nov 03 21:45:32.746398 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:32.746352 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-lk747_0c400677-019d-4644-b45e-92057290a7cf/istio-proxy/0.log" Nov 03 21:45:34.172888 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:34.172855 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp_5fda5636-bc04-47e6-8e16-01372e832616/istio-proxy/0.log" Nov 03 21:45:34.648264 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:34.648231 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mdntw_e655053f-68d6-4b7e-a193-49de67eb9f01/payload-processing/0.log" Nov 03 21:45:34.874451 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:34.874422 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76f9745bc9-fxgzq_f5ca1597-5b32-4231-97cd-e041e4ff6cd4/router/0.log" Nov 03 21:45:36.052325 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:36.052294 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5vmsj_87927855-684d-4f9a-8614-dd9649b8af0b/main/0.log" Nov 03 21:45:36.582267 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:36.582236 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qr779_b476286d-e752-4bdc-9441-5518c4860345/main/0.log" Nov 03 21:45:37.357072 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:37.357045 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps_3ab4bbe3-d96e-4e91-9345-d76342e8dc9b/main/0.log" Nov 03 21:45:44.150067 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:45:44.150033 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:45:48.092991 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:48.092955 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-9nk67_5bd2c162-bfec-4ee1-8b7c-ee3418b07532/manager/0.log" Nov 03 21:45:48.772433 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:48.772401 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcb8dd554-dczcz_4166cdea-dbd8-4a99-b275-ece179b45b9a/manager/0.log" Nov 03 21:45:49.571089 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:49.571056 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kqjhf_628e6e07-734c-4a91-81e4-21c61545b56b/proxy/0.log" Nov 03 21:45:50.340558 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:50.340522 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bbfc75f54-szn2g_c27eed1d-4d36-4d6b-a264-9217a193a291/maas-api/0.log" Nov 03 21:45:51.285597 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:51.285566 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2h456_084ba8dc-5495-4217-86b3-5b253ce4d3cc/postgres/0.log" Nov 03 21:45:52.849211 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:52.849099 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdz7m_b3bde11f-a135-4286-9b8e-c9c4c1bf3863/manager/0.log" Nov 03 21:45:53.153897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:53.153866 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-j4td6_141d9b7c-0036-47c1-8299-b0f83f14026e/manager/0.log" Nov 03 21:45:54.149161 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:54.149129 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7vr97_a4eb75db-6377-44f0-8c02-9ed350b94153/manager/0.log" Nov 03 21:45:54.830572 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:54.830539 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9gkhkg_f13d4fa7-4c1e-42fe-9ad5-d6c82f84065e/istio-proxy/0.log" Nov 03 21:45:55.056482 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:55.056458 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-4zt6g_f5b16b74-0aeb-4ff9-b150-d9f8b3693d02/discovery/0.log" Nov 03 21:45:55.724407 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:45:55.724358 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-45kq6_82930fab-d2aa-49e9-b83e-bcbc34c20ba5/istio-proxy/0.log" Nov 03 21:46:01.818314 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:01.818277 5932 scope.go:122] "RemoveContainer" containerID="302cb658d1ecec0f73970f0582f93805a4ee4ad84522ec3c035f41d7b71fd6b9" Nov 03 21:46:01.827520 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:01.827501 5932 scope.go:122] "RemoveContainer" containerID="d44f0e2170d2f631a095275317be0abb9426b6634a9a769aac1d1631e7620af5" Nov 03 21:46:06.271303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:06.271272 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-c8kfw_37560725-01f3-4f64-a350-31463f2bb770/istio-proxy/0.log" Nov 03 21:46:06.842897 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:06.842872 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-lk747_0c400677-019d-4644-b45e-92057290a7cf/istio-proxy/0.log" Nov 03 21:46:08.304459 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:08.304408 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-4xvrp_5fda5636-bc04-47e6-8e16-01372e832616/istio-proxy/0.log" Nov 03 21:46:08.810640 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:08.810598 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mdntw_e655053f-68d6-4b7e-a193-49de67eb9f01/payload-processing/0.log" Nov 03 21:46:09.043370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:09.043338 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76f9745bc9-fxgzq_f5ca1597-5b32-4231-97cd-e041e4ff6cd4/router/0.log" Nov 03 21:46:09.872224 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:09.872197 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-5vmsj_87927855-684d-4f9a-8614-dd9649b8af0b/main/0.log" Nov 03 21:46:10.309044 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:10.309016 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-qr779_b476286d-e752-4bdc-9441-5518c4860345/main/0.log" Nov 03 21:46:11.076271 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:46:11.076245 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-mwlps_3ab4bbe3-d96e-4e91-9345-d76342e8dc9b/main/0.log" Nov 03 21:46:14.150036 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:46:14.149998 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:46:44.150906 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:46:44.150808 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:47:14.151842 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:47:14.151794 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:47:29.291835 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:47:29.291806 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcb8dd554-dczcz_4166cdea-dbd8-4a99-b275-ece179b45b9a/manager/0.log" Nov 03 21:47:29.830082 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:47:29.830053 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcb8dd554-dczcz_4166cdea-dbd8-4a99-b275-ece179b45b9a/manager/0.log" Nov 03 21:47:30.581719 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:47:30.581690 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mdntw_e655053f-68d6-4b7e-a193-49de67eb9f01/payload-processing/0.log" Nov 03 21:47:30.807963 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:47:30.807936 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mdntw_e655053f-68d6-4b7e-a193-49de67eb9f01/payload-processing/0.log" Nov 03 21:47:44.149920 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:47:44.149887 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:48:14.150474 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:48:14.150396 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:48:27.210508 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:27.210475 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcb8dd554-dczcz_4166cdea-dbd8-4a99-b275-ece179b45b9a/manager/0.log" Nov 03 21:48:27.748418 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:27.748382 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7dcb8dd554-dczcz_4166cdea-dbd8-4a99-b275-ece179b45b9a/manager/0.log" Nov 03 21:48:28.513376 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:28.513349 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mdntw_e655053f-68d6-4b7e-a193-49de67eb9f01/payload-processing/0.log" Nov 03 21:48:28.733574 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:28.733545 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-mdntw_e655053f-68d6-4b7e-a193-49de67eb9f01/payload-processing/0.log" Nov 03 21:48:38.174608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:38.174574 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5khcq_c947407f-1cc6-4c01-997e-99e19a3c139c/global-pull-secret-syncer/0.log" Nov 03 21:48:40.070608 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:40.070580 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-48l5q_a995cca6-542f-406e-a938-aa58715eac9a/connection-checker/0.log" Nov 03 21:48:40.475272 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:40.475241 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8bznz_02bb73c6-e6e2-4d77-8458-20ba459b831a/connection-checker/0.log" Nov 03 21:48:40.879647 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:40.879613 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-k2tz8_6521fc8f-d7c8-46cb-b50c-eb7ff5ec1750/connection-checker/0.log" Nov 03 21:48:41.894947 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:41.894918 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l5fnw_eee402df-6e0a-4cac-a025-b981a2ee6329/konnectivity-agent/0.log" Nov 03 21:48:43.821056 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:48:43.821025 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-52.us-west-2.compute.internal_d46efce11f8f6dc55307823b4dc3bdd4/haproxy/0.log" Nov 03 21:48:44.150104 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:48:44.150067 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:49:07.507308 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.507268 5932 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn"] Nov 03 21:49:07.508045 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.508022 5932 cpu_manager.go:414] "containerMap: removing container" podUID="9bb72260-cb01-412f-9cc5-312d97ecba96" containerName="payload-processing" Nov 03 21:49:07.508139 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.508047 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb72260-cb01-412f-9cc5-312d97ecba96" containerName="payload-processing" Nov 03 21:49:07.508139 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.508106 5932 cpu_manager.go:414] "containerMap: removing container" podUID="6b384176-2244-4a40-b7b6-c88e6e99fad6" containerName="maas-api" Nov 03 21:49:07.508139 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.508115 5932 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b384176-2244-4a40-b7b6-c88e6e99fad6" containerName="maas-api" Nov 03 21:49:07.508303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.508226 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b384176-2244-4a40-b7b6-c88e6e99fad6" containerName="maas-api" Nov 03 21:49:07.508303 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.508242 5932 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb72260-cb01-412f-9cc5-312d97ecba96" containerName="payload-processing" Nov 03 21:49:07.511510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.511488 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.549388 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.549355 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-proc\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.549563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.549408 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-podres\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.549563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.549486 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-sys\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.549563 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.549548 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8k8\" (UniqueName: \"kubernetes.io/projected/293eb61e-0df0-4142-b587-7bdbb57d81e0-kube-api-access-6v8k8\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.549784 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.549655 5932 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-lib-modules\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.563135 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.563085 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtnjt\"/\"kube-root-ca.crt\"" Nov 03 21:49:07.563135 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.563103 5932 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mtnjt\"/\"default-dockercfg-bkvpt\"" Nov 03 21:49:07.609510 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.609474 5932 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtnjt\"/\"openshift-service-ca.crt\"" Nov 03 21:49:07.612003 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.611979 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn"] Nov 03 21:49:07.650535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650493 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-podres\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650535 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650546 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-sys\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650601 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8k8\" (UniqueName: \"kubernetes.io/projected/293eb61e-0df0-4142-b587-7bdbb57d81e0-kube-api-access-6v8k8\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650665 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-lib-modules\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650664 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-podres\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650694 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-sys\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650754 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-lib-modules\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.650823 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650764 5932 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-proc\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.651087 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.650831 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/293eb61e-0df0-4142-b587-7bdbb57d81e0-proc\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.707498 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.707466 5932 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8k8\" (UniqueName: \"kubernetes.io/projected/293eb61e-0df0-4142-b587-7bdbb57d81e0-kube-api-access-6v8k8\") pod \"perf-node-gather-daemonset-c8lfn\" (UID: \"293eb61e-0df0-4142-b587-7bdbb57d81e0\") " pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.824406 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.824299 5932 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:07.885387 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:07.885356 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zdz7m_b3bde11f-a135-4286-9b8e-c9c4c1bf3863/manager/0.log" Nov 03 21:49:08.202960 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:08.202878 5932 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn"] Nov 03 21:49:08.211528 ip-10-0-138-52 kubenswrapper[5932]: W1103 21:49:08.211496 5932 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod293eb61e_0df0_4142_b587_7bdbb57d81e0.slice/crio-d703c3d603314eb142eaf4be44da9f37f1d10edd4cc49d547d0bf2e37b1db016 WatchSource:0}: Error finding container d703c3d603314eb142eaf4be44da9f37f1d10edd4cc49d547d0bf2e37b1db016: Status 404 returned error can't find the container with id d703c3d603314eb142eaf4be44da9f37f1d10edd4cc49d547d0bf2e37b1db016 Nov 03 21:49:08.287292 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:08.287255 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-j4td6_141d9b7c-0036-47c1-8299-b0f83f14026e/manager/0.log" Nov 03 21:49:08.451460 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:08.451421 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" event={"ID":"293eb61e-0df0-4142-b587-7bdbb57d81e0","Type":"ContainerStarted","Data":"65eae2517a3f003aa13fbe4b0cfac844d2fba02b4b087da2e9f0a824e391df5c"} Nov 03 21:49:08.451692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:08.451485 5932 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" event={"ID":"293eb61e-0df0-4142-b587-7bdbb57d81e0","Type":"ContainerStarted","Data":"d703c3d603314eb142eaf4be44da9f37f1d10edd4cc49d547d0bf2e37b1db016"} Nov 03 21:49:08.451692 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:08.451528 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:08.560939 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:08.560838 5932 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" podStartSLOduration=1.560822854 podStartE2EDuration="1.560822854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:49:08.559055801 +0000 UTC m=+4756.624297275" watchObservedRunningTime="2026-11-03 21:49:08.560822854 +0000 UTC m=+4756.626064328" Nov 03 21:49:10.257404 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:10.257371 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-7vr97_a4eb75db-6377-44f0-8c02-9ed350b94153/manager/0.log" Nov 03 21:49:14.150806 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:49:14.150768 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:49:14.467566 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:14.467497 5932 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mtnjt/perf-node-gather-daemonset-c8lfn" Nov 03 21:49:16.013543 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:16.013511 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-627kh_6b4895c8-b425-4009-a06b-47f1d51b8784/cluster-monitoring-operator/0.log" Nov 03 21:49:21.072607 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:21.072575 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqx4f_aefd9a58-f889-4651-9b0b-f7bfe254355f/node-exporter/0.log" Nov 03 21:49:21.475035 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:21.475007 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqx4f_aefd9a58-f889-4651-9b0b-f7bfe254355f/kube-rbac-proxy/0.log" Nov 03 21:49:21.878273 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:21.878250 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqx4f_aefd9a58-f889-4651-9b0b-f7bfe254355f/init-textfile/0.log" Nov 03 21:49:34.108327 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:34.108208 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-2b69b_5bc681f0-27c6-4cad-8bf6-82835e009976/networking-console-plugin/0.log" Nov 03 21:49:36.789488 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:36.789451 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:49:36.793913 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:36.793888 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/2.log" Nov 03 21:49:42.270370 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:42.270344 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-b74gt_6ab99b87-827c-4aba-a47e-b86a847ca383/volume-data-source-validator/0.log" Nov 03 21:49:44.149467 ip-10-0-138-52 kubenswrapper[5932]: E1103 21:49:44.149432 5932 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-dpsrk" containerName="extractor" Nov 03 21:49:49.204180 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:49.204150 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sd2tb_e44f243f-ff0c-4e9a-9a91-b0079c2667b6/dns/0.log" Nov 03 21:49:49.604336 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:49.604263 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sd2tb_e44f243f-ff0c-4e9a-9a91-b0079c2667b6/kube-rbac-proxy/0.log" Nov 03 21:49:50.423339 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:50.423307 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cwd62_f34740ae-e26e-4188-a405-4da96b57553e/dns-node-resolver/0.log" Nov 03 21:49:54.070600 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:54.070567 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2j4jr_2ab15921-e819-4567-af4a-9d3684c054ae/node-ca/0.log" Nov 03 21:49:56.457787 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:56.457747 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log" Nov 03 21:49:56.473299 ip-10-0-138-52 kubenswrapper[5932]: I1103 21:49:56.473278 5932 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-ljjts_c7a59cf1-454d-4019-9e05-0108e38b8725/console-operator/1.log"